dbcli/sqlglot
1
0
Fork 0
Code Issues Activity

Merging upstream version 23.16.0.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-13 21:35:32 +01:00
parent d0f42f708a
commit 213191b8e3
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
93 changed files with 64106 additions and 59061 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.github/workflows/python-package.yml vendored
View file

@ -10,7 +10,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}

4
.github/workflows/python-publish.yml vendored
View file

@ -12,7 +12,7 @@ jobs:
should-deploy-rs:
runs-on: ubuntu-latest
outputs:
deploy: ${{ steps.set_deploy.outputs.deploy }}
deploy: ${{ steps.check_deploy.outputs.deploy }}
steps:
- uses: actions/checkout@v3
- id: check_deploy
@ -47,7 +47,7 @@ jobs:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
python-version: '3.11'
architecture: ${{ matrix.python-architecture || 'x64' }}
- name: Build wheels
uses: PyO3/maturin-action@v1

186
CHANGELOG.md
View file

@ -1,6 +1,181 @@
Changelog
=========
## [v23.15.10] - 2024-05-17
### :sparkles: New Features
- [`89c1d3a`](https://github.com/tobymao/sqlglot/commit/89c1d3a4dd3387576c384413b3a8991a2dd030de) - **clickhouse**: support generate TimestampTrunc, Variance, Stddev *(PR [#3489](https://github.com/tobymao/sqlglot/pull/3489) by [@longxiaofei](https://github.com/longxiaofei))*
### :bug: Bug Fixes
- [`03879bb`](https://github.com/tobymao/sqlglot/commit/03879bb3249ee83cce34d629f1016575d3b932e3) - **postgres**: date_trunc supports time zone *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`6e7f37a`](https://github.com/tobymao/sqlglot/commit/6e7f37af86a4f36ec47ea4ef3519e5c97376e090) - copy into pretty printing and default dialect *(PR [#3496](https://github.com/tobymao/sqlglot/pull/3496) by [@tobymao](https://github.com/tobymao))*
- [`e8600e2`](https://github.com/tobymao/sqlglot/commit/e8600e24370a131a0b375a1a9943fdf590968198) - property eq needs highest precedence *(commit by [@tobymao](https://github.com/tobymao))*
## [v23.15.9] - 2024-05-17
### :boom: BREAKING CHANGES
- due to [`846d5cd`](https://github.com/tobymao/sqlglot/commit/846d5cd2fe85f836f5ad888e783fedfa2108d579) - set default precision / width for DECIMAL type *(PR [#3472](https://github.com/tobymao/sqlglot/pull/3472) by [@georgesittas](https://github.com/georgesittas))*:
set default precision / width for DECIMAL type (#3472)
- due to [`e3ff67b`](https://github.com/tobymao/sqlglot/commit/e3ff67b0327a217a0523f82e6a11940feab1a8ac) - preserve star clauses (EXCLUDE, RENAME, REPLACE) *(PR [#3477](https://github.com/tobymao/sqlglot/pull/3477) by [@georgesittas](https://github.com/georgesittas))*:
preserve star clauses (EXCLUDE, RENAME, REPLACE) (#3477)
- due to [`b417c80`](https://github.com/tobymao/sqlglot/commit/b417c80b4208df1b97363db53af42158aa97bbd6) - parse TININT into UTINYINT to improve transpilation *(PR [#3486](https://github.com/tobymao/sqlglot/pull/3486) by [@georgesittas](https://github.com/georgesittas))*:
parse TININT into UTINYINT to improve transpilation (#3486)
- due to [`54e31af`](https://github.com/tobymao/sqlglot/commit/54e31af7d86138662c9619d50b4ae2e68e04942b) - add DECLARE statement parsing *(PR [#3462](https://github.com/tobymao/sqlglot/pull/3462) by [@jlucas-fsp](https://github.com/jlucas-fsp))*:
add DECLARE statement parsing (#3462)
- due to [`7287bb9`](https://github.com/tobymao/sqlglot/commit/7287bb9bf578b2b3afaf25647f505b9d73040dc7) - nested cte ordering closes [#3488](https://github.com/tobymao/sqlglot/pull/3488) *(commit by [@tobymao](https://github.com/tobymao))*:
nested cte ordering closes #3488
### :sparkles: New Features
- [`2c29bf3`](https://github.com/tobymao/sqlglot/commit/2c29bf3b7a163b88754c4593996bbba9b3c791b6) - **snowflake**: add support for CREATE TAG DDL statement *(PR [#3473](https://github.com/tobymao/sqlglot/pull/3473) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *addresses issue [#3468](https://github.com/tobymao/sqlglot/issues/3468) opened by [@tekumara](https://github.com/tekumara)*
- [`2433993`](https://github.com/tobymao/sqlglot/commit/24339934167e651f2afd6966024e4d96ef55c677) - **transpiler**: handle different hex behavior for dialects *(PR [#3463](https://github.com/tobymao/sqlglot/pull/3463) by [@viplazylmht](https://github.com/viplazylmht))*
- :arrow_lower_right: *addresses issue [#3460](https://github.com/tobymao/sqlglot/issues/3460) opened by [@viplazylmht](https://github.com/viplazylmht)*
- [`0009e09`](https://github.com/tobymao/sqlglot/commit/0009e09b1a7f94f85985670a09bb0be92c673b46) - add epoch_ms of duckdb to other dialects *(PR [#3471](https://github.com/tobymao/sqlglot/pull/3471) by [@longxiaofei](https://github.com/longxiaofei))*
- [`461215b`](https://github.com/tobymao/sqlglot/commit/461215b259de98125ea6b09d7bd875edb3ccce75) - **clickhouse**: add support for PROJECTION in CREATE TABLE statement *(PR [#3465](https://github.com/tobymao/sqlglot/pull/3465) by [@GaliFFun](https://github.com/GaliFFun))*
- [`54e31af`](https://github.com/tobymao/sqlglot/commit/54e31af7d86138662c9619d50b4ae2e68e04942b) - **tsql**: add DECLARE statement parsing *(PR [#3462](https://github.com/tobymao/sqlglot/pull/3462) by [@jlucas-fsp](https://github.com/jlucas-fsp))*
- [`c811adb`](https://github.com/tobymao/sqlglot/commit/c811adb73e6f83265fedc26274c7d4b40f8a1c85) - snowflake array_construct_compact to spark *(commit by [@tobymao](https://github.com/tobymao))*
### :bug: Bug Fixes
- [`58d5f2b`](https://github.com/tobymao/sqlglot/commit/58d5f2bece42acdda5f8c08d30e6f61a5e538d4c) - **presto**: fix parsing and generating hash functions presto/trino *(PR [#3459](https://github.com/tobymao/sqlglot/pull/3459) by [@viplazylmht](https://github.com/viplazylmht))*
- :arrow_lower_right: *fixes issue [#3458](https://github.com/tobymao/sqlglot/issues/3458) opened by [@viplazylmht](https://github.com/viplazylmht)*
- [`065281e`](https://github.com/tobymao/sqlglot/commit/065281e28be75597f3f97cee22995423ed483660) - **optimizer**: fix multiple bugs in unnest_subqueries, clean up test suite *(PR [#3464](https://github.com/tobymao/sqlglot/pull/3464) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3448](https://github.com/tobymao/sqlglot/issues/3448) opened by [@yesemsanthoshkumar](https://github.com/yesemsanthoshkumar)*
- [`80ba1e8`](https://github.com/tobymao/sqlglot/commit/80ba1e8786a6347b8f20f340c185a0b41d017c73) - preserve quotes for projections produced by the eliminate_qualify rule *(PR [#3470](https://github.com/tobymao/sqlglot/pull/3470) by [@aersam](https://github.com/aersam))*
- :arrow_lower_right: *fixes issue [#3467](https://github.com/tobymao/sqlglot/issues/3467) opened by [@aersam](https://github.com/aersam)*
- [`3bc1fbe`](https://github.com/tobymao/sqlglot/commit/3bc1fbed40d9d0d05f189ca60fdc7af19b815e8b) - make quoting of alias_or_name in eliminate_qualify more robust *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`1843e9b`](https://github.com/tobymao/sqlglot/commit/1843e9b825da6e97bda8c7b4fffce40baf199af1) - allow parameters in user-defined types *(PR [#3474](https://github.com/tobymao/sqlglot/pull/3474) by [@georgesittas](https://github.com/georgesittas))*
- [`e004d2a`](https://github.com/tobymao/sqlglot/commit/e004d2a3d88ea77d34ecdb8290df1e73511e6b6c) - **duckdb**: preserve precedence of json extraction when converting to arrow syntax *(PR [#3478](https://github.com/tobymao/sqlglot/pull/3478) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3476](https://github.com/tobymao/sqlglot/issues/3476) opened by [@asiunov](https://github.com/asiunov)*
- [`e3ff67b`](https://github.com/tobymao/sqlglot/commit/e3ff67b0327a217a0523f82e6a11940feab1a8ac) - **snowflake**: preserve star clauses (EXCLUDE, RENAME, REPLACE) *(PR [#3477](https://github.com/tobymao/sqlglot/pull/3477) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3475](https://github.com/tobymao/sqlglot/issues/3475) opened by [@asiunov](https://github.com/asiunov)*
- [`428fd61`](https://github.com/tobymao/sqlglot/commit/428fd61574e10be9afab23ac711758b229cc174f) - **mysql**: generate CONCAT for DPipe *(PR [#3482](https://github.com/tobymao/sqlglot/pull/3482) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3481](https://github.com/tobymao/sqlglot/issues/3481) opened by [@Toms1999](https://github.com/Toms1999)*
- [`b417c80`](https://github.com/tobymao/sqlglot/commit/b417c80b4208df1b97363db53af42158aa97bbd6) - **tsql**: parse TININT into UTINYINT to improve transpilation *(PR [#3486](https://github.com/tobymao/sqlglot/pull/3486) by [@georgesittas](https://github.com/georgesittas))*
- [`a3ff49e`](https://github.com/tobymao/sqlglot/commit/a3ff49e93f2c6752f512192ca8b6b6ad18fc925a) - **presto**: fix DELETE DML statement for presto/trino *(PR [#3466](https://github.com/tobymao/sqlglot/pull/3466) by [@viplazylmht](https://github.com/viplazylmht))*
- [`7287bb9`](https://github.com/tobymao/sqlglot/commit/7287bb9bf578b2b3afaf25647f505b9d73040dc7) - nested cte ordering closes [#3488](https://github.com/tobymao/sqlglot/pull/3488) *(commit by [@tobymao](https://github.com/tobymao))*
- [`5b64475`](https://github.com/tobymao/sqlglot/commit/5b64475bfd2d6a0ddcb3d0adb60d06dca62421a0) - allow rollup to be used as an identifier *(PR [#3495](https://github.com/tobymao/sqlglot/pull/3495) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3494](https://github.com/tobymao/sqlglot/issues/3494) opened by [@dangoldin](https://github.com/dangoldin)*
### :recycle: Refactors
- [`846d5cd`](https://github.com/tobymao/sqlglot/commit/846d5cd2fe85f836f5ad888e783fedfa2108d579) - **snowflake**: set default precision / width for DECIMAL type *(PR [#3472](https://github.com/tobymao/sqlglot/pull/3472) by [@georgesittas](https://github.com/georgesittas))*
- [`930f923`](https://github.com/tobymao/sqlglot/commit/930f923c6da182be33ad4c912b64ec052a63af30) - clean up Hex / LowerHex implementation *(PR [#3483](https://github.com/tobymao/sqlglot/pull/3483) by [@georgesittas](https://github.com/georgesittas))*
- [`883fcd7`](https://github.com/tobymao/sqlglot/commit/883fcd78645539a275b66472f0bd1dfe1d3d4401) - **presto**: make DELETE transpilation more robust *(PR [#3487](https://github.com/tobymao/sqlglot/pull/3487) by [@georgesittas](https://github.com/georgesittas))*
- [`49f7f85`](https://github.com/tobymao/sqlglot/commit/49f7f857634ae85547c805ac53911895407dd7cb) - **tsql**: handle TABLE <schema> more gracefully for DeclareItem *(commit by [@georgesittas](https://github.com/georgesittas))*
## [v23.15.8] - 2024-05-11
### :boom: BREAKING CHANGES
- due to [`510f8b5`](https://github.com/tobymao/sqlglot/commit/510f8b5726c59a13284e9482dc47d488559e6c9e) - improve transpilation of TABLESAMPLE clause *(PR [#3457](https://github.com/tobymao/sqlglot/pull/3457) by [@georgesittas](https://github.com/georgesittas))*:
improve transpilation of TABLESAMPLE clause (#3457)
### :sparkles: New Features
- [`510f8b5`](https://github.com/tobymao/sqlglot/commit/510f8b5726c59a13284e9482dc47d488559e6c9e) - improve transpilation of TABLESAMPLE clause *(PR [#3457](https://github.com/tobymao/sqlglot/pull/3457) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *addresses issue [#3456](https://github.com/tobymao/sqlglot/issues/3456) opened by [@whummer](https://github.com/whummer)*
- [`e28c959`](https://github.com/tobymao/sqlglot/commit/e28c959bf44208bdb3821b38c13fde59f1944fbb) - make create table cmd parsing less aggressive so that they can be used in sqlmesh @if macros *(commit by [@tobymao](https://github.com/tobymao))*
## [v23.15.7] - 2024-05-11
### :wrench: Chores
- [`c3bb3da`](https://github.com/tobymao/sqlglot/commit/c3bb3da670d06cb2eef545a909635224b6e7c205) - change python-version to 3.11 for build-rs *(commit by [@georgesittas](https://github.com/georgesittas))*
## [v23.15.6] - 2024-05-11
### :wrench: Chores
- [`cd8f568`](https://github.com/tobymao/sqlglot/commit/cd8f568dba53efe6b9883035c48a67134016e612) - fix rust deployment workflow bug *(commit by [@georgesittas](https://github.com/georgesittas))*
## [v23.15.3] - 2024-05-10
### :wrench: Chores
- [`130255e`](https://github.com/tobymao/sqlglot/commit/130255ebc927c48b3d3e479e17c38269bd7d8056) - update rust *(commit by [@tobymao](https://github.com/tobymao))*
## [v23.15.2] - 2024-05-10
### :sparkles: New Features
- [`116172a`](https://github.com/tobymao/sqlglot/commit/116172a41119e72aaf618a83761f73d52f0440d2) - add support for ON property in ALTER and DROP statements *(PR [#3450](https://github.com/tobymao/sqlglot/pull/3450) by [@GaliFFun](https://github.com/GaliFFun))*
- [`aa104fd`](https://github.com/tobymao/sqlglot/commit/aa104fd2ccd73a13ca60fa3de3296ed4c007e8da) - add semi colon comments *(commit by [@tobymao](https://github.com/tobymao))*
### :bug: Bug Fixes
- [`2c62267`](https://github.com/tobymao/sqlglot/commit/2c62267e2ae908d10d8164f080bc66a133596bf6) - **bigquery**: fix SHA1 generator *(PR [#3453](https://github.com/tobymao/sqlglot/pull/3453) by [@viplazylmht](https://github.com/viplazylmht))*
- :arrow_lower_right: *fixes issue [#3451](https://github.com/tobymao/sqlglot/issues/3451) opened by [@viplazylmht](https://github.com/viplazylmht)*
- [`fb3dea9`](https://github.com/tobymao/sqlglot/commit/fb3dea9a803157b4684cd62e2ef0b6a6b612f7e1) - **clickhouse**: fix parsing and generating hash functions *(PR [#3454](https://github.com/tobymao/sqlglot/pull/3454) by [@viplazylmht](https://github.com/viplazylmht))*
- :arrow_lower_right: *fixes issue [#3452](https://github.com/tobymao/sqlglot/issues/3452) opened by [@viplazylmht](https://github.com/viplazylmht)*
- [`b76dfda`](https://github.com/tobymao/sqlglot/commit/b76dfda7b4122a59c52bcbb445cffc6617e68b8c) - **snowflake**: COPY Subquery postfix *(PR [#3449](https://github.com/tobymao/sqlglot/pull/3449) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *fixes issue [#3434](https://github.com/tobymao/sqlglot/issues/3434) opened by [@whummer](https://github.com/whummer)*
### :wrench: Chores
- [`684df5f`](https://github.com/tobymao/sqlglot/commit/684df5f7e11bb89def9ff71da0913de222bdaf3c) - remove unnecessary set_op *(commit by [@tobymao](https://github.com/tobymao))*
## [v23.15.1] - 2024-05-10
### :bug: Bug Fixes
- [`33ac4fc`](https://github.com/tobymao/sqlglot/commit/33ac4fca3e5f162500ddde529cd69c338a6fecc5) - add create view tsql *(commit by [@tobymao](https://github.com/tobymao))*
## [v23.15.0] - 2024-05-09
### :boom: BREAKING CHANGES
- due to [`9338ebc`](https://github.com/tobymao/sqlglot/commit/9338ebc6dc9635f12639b562ee2af140cf708b6b) - tsql drop view no catalog *(commit by [@tobymao](https://github.com/tobymao))*:
tsql drop view no catalog
### :sparkles: New Features
- [`80670bb`](https://github.com/tobymao/sqlglot/commit/80670bbd1e062cc476dcee17d0b9972ff7dc0424) - **snowflake**: Support for APPROX_PERCENTILE *(PR [#3426](https://github.com/tobymao/sqlglot/pull/3426) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *addresses issue [#3424](https://github.com/tobymao/sqlglot/issues/3424) opened by [@baruchoxman](https://github.com/baruchoxman)*
- [`b46c5b3`](https://github.com/tobymao/sqlglot/commit/b46c5b3ddaed359fb59264f00d7033c7b36bd9a4) - **clickhouse**: add support for partition expression *(PR [#3428](https://github.com/tobymao/sqlglot/pull/3428) by [@GaliFFun](https://github.com/GaliFFun))*
- [`07badc9`](https://github.com/tobymao/sqlglot/commit/07badc9d155cfd6d0c70e4419ed763b8c52b4973) - **clickhouse**: add support for ALTER TABLE REPLACE PARTITION statement *(PR [#3441](https://github.com/tobymao/sqlglot/pull/3441) by [@GaliFFun](https://github.com/GaliFFun))*
- [`baf39e7`](https://github.com/tobymao/sqlglot/commit/baf39e78cdebf5478b59f83120c43b39b27d1a31) - **redshift**: improve ALTER TABLE .. ALTER .. support *(PR [#3444](https://github.com/tobymao/sqlglot/pull/3444) by [@georgesittas](https://github.com/georgesittas))*
### :bug: Bug Fixes
- [`e8014e2`](https://github.com/tobymao/sqlglot/commit/e8014e2a479c37ef75510e7d5ca90ed30522ce60) - **mysql**: Parse REPLACE statement as Command *(PR [#3425](https://github.com/tobymao/sqlglot/pull/3425) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *fixes issue [#3423](https://github.com/tobymao/sqlglot/issues/3423) opened by [@DyCheer](https://github.com/DyCheer)*
- [`273731f`](https://github.com/tobymao/sqlglot/commit/273731fd8cba4d6bda0d7ce109f25c49de0ec95c) - **snowflake**: parse CREATE SEQUENCE with commas *(PR [#3436](https://github.com/tobymao/sqlglot/pull/3436) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3435](https://github.com/tobymao/sqlglot/issues/3435) opened by [@whummer](https://github.com/whummer)*
- [`761ba6f`](https://github.com/tobymao/sqlglot/commit/761ba6fb507158d4e5ea51ca396809be91c11ebf) - don't generate connector comments when comments=False closes [#3439](https://github.com/tobymao/sqlglot/pull/3439) *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`a2a6eaa`](https://github.com/tobymao/sqlglot/commit/a2a6eaa5d7ace2879ded7c3a4cf4192b75c07f26) - handle empty string in connector comment padding *(PR [#3437](https://github.com/tobymao/sqlglot/pull/3437) by [@uncledata](https://github.com/uncledata))*
- [`1bc0ce5`](https://github.com/tobymao/sqlglot/commit/1bc0ce57eca5e401a4c39237b52ee722bdfb46af) - func to binary MOD generation *(PR [#3440](https://github.com/tobymao/sqlglot/pull/3440) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *fixes issue [#3431](https://github.com/tobymao/sqlglot/issues/3431) opened by [@daniel769](https://github.com/daniel769)*
- [`5cfb29c`](https://github.com/tobymao/sqlglot/commit/5cfb29c7ff6015e39d7fd5b94ed2aa66436e33ae) - **bigquery**: MOD edge case *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`9825c4c`](https://github.com/tobymao/sqlglot/commit/9825c4cb616af07a048109c499666081bc6e4eba) - improve error handling for nested schema levels *(PR [#3445](https://github.com/tobymao/sqlglot/pull/3445) by [@tobymao](https://github.com/tobymao))*
- [`c309def`](https://github.com/tobymao/sqlglot/commit/c309defa450f755dbed1d1b6f276b4b1765166e2) - **duckdb**: use name sequence instead of single _t for unnest alias *(PR [#3446](https://github.com/tobymao/sqlglot/pull/3446) by [@georgesittas](https://github.com/georgesittas))*
- [`0927ae3`](https://github.com/tobymao/sqlglot/commit/0927ae3c448ebf068b89bfa5e46b8f135121b470) - **executor**: use timezone-aware object to represent datetime in UTC *(PR [#3447](https://github.com/tobymao/sqlglot/pull/3447) by [@georgesittas](https://github.com/georgesittas))*
- [`9338ebc`](https://github.com/tobymao/sqlglot/commit/9338ebc6dc9635f12639b562ee2af140cf708b6b) - tsql drop view no catalog *(commit by [@tobymao](https://github.com/tobymao))*
### :wrench: Chores
- [`30f9d30`](https://github.com/tobymao/sqlglot/commit/30f9d30d8ab3727a43b1e6f363f28631cbfa7f92) - bump ruff to 0.4.3 *(PR [#3430](https://github.com/tobymao/sqlglot/pull/3430) by [@georgesittas](https://github.com/georgesittas))*
- [`91bed56`](https://github.com/tobymao/sqlglot/commit/91bed5607e442d416021a1f93e4a457fb47b6a1f) - test 3.12 *(commit by [@tobymao](https://github.com/tobymao))*
## [v23.14.0] - 2024-05-07
### :boom: BREAKING CHANGES
- due to [`258ad3b`](https://github.com/tobymao/sqlglot/commit/258ad3bbf73f55d02ed78a93fa0f16d4630159e3) - parse column instead of identifier for SET assignment LHS *(PR [#3417](https://github.com/tobymao/sqlglot/pull/3417) by [@georgesittas](https://github.com/georgesittas))*:
parse column instead of identifier for SET assignment LHS (#3417)
### :bug: Bug Fixes
- [`258ad3b`](https://github.com/tobymao/sqlglot/commit/258ad3bbf73f55d02ed78a93fa0f16d4630159e3) - parse column instead of identifier for SET assignment LHS *(PR [#3417](https://github.com/tobymao/sqlglot/pull/3417) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3415](https://github.com/tobymao/sqlglot/issues/3415) opened by [@tekumara](https://github.com/tekumara)*
- [`17c31da`](https://github.com/tobymao/sqlglot/commit/17c31da9e159dc1cdd91bd6df38c43606bdc48c9) - **lineage**: get rid of comments in Node names *(PR [#3418](https://github.com/tobymao/sqlglot/pull/3418) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3414](https://github.com/tobymao/sqlglot/issues/3414) opened by [@jaspev123](https://github.com/jaspev123)*
- [`ea197ea`](https://github.com/tobymao/sqlglot/commit/ea197eae2fcdbeba395b53cf4864fc2e44134c71) - **snowflake**: ensure OBJECT_CONSTRUCT is not generated inside of VALUES *(PR [#3419](https://github.com/tobymao/sqlglot/pull/3419) by [@georgesittas](https://github.com/georgesittas))*
## [v23.13.7] - 2024-05-04
### :wrench: Chores
- [`4dbcd4f`](https://github.com/tobymao/sqlglot/commit/4dbcd4f7147204b7bafa32d14dfe615882562b6b) - refactor publish workflow for sqlglotrs releasing *(commit by [@georgesittas](https://github.com/georgesittas))*
## [v23.13.6] - 2024-05-04
### :wrench: Chores
- [`aa4f90a`](https://github.com/tobymao/sqlglot/commit/aa4f90acde9c022fb7f984b30763c732977c1b4c) - refactor publish workflow for sqlglotrs releasing *(commit by [@georgesittas](https://github.com/georgesittas))*
@ -3482,3 +3657,14 @@ Changelog
[v23.13.4]: https://github.com/tobymao/sqlglot/compare/v23.13.3...v23.13.4
[v23.13.5]: https://github.com/tobymao/sqlglot/compare/v23.13.4...v23.13.5
[v23.13.6]: https://github.com/tobymao/sqlglot/compare/v23.13.5...v23.13.6
[v23.13.7]: https://github.com/tobymao/sqlglot/compare/v23.13.6...v23.13.7
[v23.14.0]: https://github.com/tobymao/sqlglot/compare/v23.13.7...v23.14.0
[v23.15.0]: https://github.com/tobymao/sqlglot/compare/v23.14.0...v23.15.0
[v23.15.1]: https://github.com/tobymao/sqlglot/compare/v23.15.0...v23.15.1
[v23.15.2]: https://github.com/tobymao/sqlglot/compare/v23.15.1...v23.15.2
[v23.15.3]: https://github.com/tobymao/sqlglot/compare/v23.15.2...v23.15.3
[v23.15.6]: https://github.com/tobymao/sqlglot/compare/v23.15.5...v23.15.6
[v23.15.7]: https://github.com/tobymao/sqlglot/compare/v23.15.6...v23.15.7
[v23.15.8]: https://github.com/tobymao/sqlglot/compare/v23.15.7...v23.15.8
[v23.15.9]: https://github.com/tobymao/sqlglot/compare/v23.15.8...v23.15.9
[v23.15.10]: https://github.com/tobymao/sqlglot/compare/v23.15.9...v23.15.10

2
docs/search.js
View file

File diff suppressed because one or more lines are too long

8
docs/sqlglot/_version.html
View file

@ -76,8 +76,8 @@
</span><span id="L-12"><a href="#L-12"><span class="linenos">12</span></a><span class="n">__version_tuple__</span><span class="p">:</span> <span class="n">VERSION_TUPLE</span>
</span><span id="L-13"><a href="#L-13"><span class="linenos">13</span></a><span class="n">version_tuple</span><span class="p">:</span> <span class="n">VERSION_TUPLE</span>
</span><span id="L-14"><a href="#L-14"><span class="linenos">14</span></a>
</span><span id="L-15"><a href="#L-15"><span class="linenos">15</span></a><span class="n">__version__</span> <span class="o">=</span> <span class="n">version</span> <span class="o">=</span> <span class="s1">&#39;23.13.6&#39;</span>
</span><span id="L-16"><a href="#L-16"><span class="linenos">16</span></a><span class="n">__version_tuple__</span> <span class="o">=</span> <span class="n">version_tuple</span> <span class="o">=</span> <span class="p">(</span><span class="mi">23</span><span class="p">,</span> <span class="mi">13</span><span class="p">,</span> <span class="mi">6</span><span class="p">)</span>
</span><span id="L-15"><a href="#L-15"><span class="linenos">15</span></a><span class="n">__version__</span> <span class="o">=</span> <span class="n">version</span> <span class="o">=</span> <span class="s1">&#39;23.15.10&#39;</span>
</span><span id="L-16"><a href="#L-16"><span class="linenos">16</span></a><span class="n">__version_tuple__</span> <span class="o">=</span> <span class="n">version_tuple</span> <span class="o">=</span> <span class="p">(</span><span class="mi">23</span><span class="p">,</span> <span class="mi">15</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span>
</span></pre></div>
@ -97,7 +97,7 @@
<section id="version">
<div class="attr variable">
<span class="name">version</span><span class="annotation">: str</span> =
<span class="default_value">&#39;23.13.6&#39;</span>
<span class="default_value">&#39;23.15.10&#39;</span>
</div>
@ -109,7 +109,7 @@
<section id="version_tuple">
<div class="attr variable">
<span class="name">version_tuple</span><span class="annotation">: object</span> =
<span class="default_value">(23, 13, 6)</span>
<span class="default_value">(23, 15, 10)</span>
</div>

40
docs/sqlglot/dataframe/sql.html
View file

@ -791,7 +791,7 @@
<div class="attr function">
<span class="def">def</span>
<span class="name">createDataFrame</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">data</span><span class="p">:</span> <span class="n">Sequence</span><span class="p">[</span><span class="n">Union</span><span class="p">[</span><span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980125852944&#39;</span><span class="o">&gt;</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980125852944&#39;</span><span class="o">&gt;</span><span class="p">],</span> <span class="n">Tuple</span><span class="p">]]</span>,</span><span class="param"> <span class="n">schema</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980125405248&#39;</span><span class="o">&gt;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">samplingRatio</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">float</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">verifySchema</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span></span><span class="return-annotation">) -> <span class="n"><a href="#DataFrame">DataFrame</a></span>:</span></span>
<span class="name">createDataFrame</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">data</span><span class="p">:</span> <span class="n">Sequence</span><span class="p">[</span><span class="n">Union</span><span class="p">[</span><span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015153824&#39;</span><span class="o">&gt;</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015153824&#39;</span><span class="o">&gt;</span><span class="p">],</span> <span class="n">Tuple</span><span class="p">]]</span>,</span><span class="param"> <span class="n">schema</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015110096&#39;</span><span class="o">&gt;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">samplingRatio</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">float</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">verifySchema</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span></span><span class="return-annotation">) -> <span class="n"><a href="#DataFrame">DataFrame</a></span>:</span></span>
<label class="view-source-button" for="SparkSession.createDataFrame-view-source"><span>View Source</span></label>
@ -1848,7 +1848,7 @@
<input id="DataFrame.__init__-view-source" class="view-source-toggle-state" type="checkbox" aria-hidden="true" tabindex="-1">
<div class="attr function">
<span class="name">DataFrame</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">spark</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980128668496&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="n">expression</span><span class="p">:</span> <span class="n"><a href="../expressions.html#Select">sqlglot.expressions.Select</a></span>,</span><span class="param"> <span class="n">branch_id</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">sequence_id</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">last_op</span><span class="p">:</span> <span class="n">sqlglot</span><span class="o">.</span><span class="n">dataframe</span><span class="o">.</span><span class="n">sql</span><span class="o">.</span><span class="n">operations</span><span class="o">.</span><span class="n">Operation</span> <span class="o">=</span> <span class="o">&lt;</span><span class="n">Operation</span><span class="o">.</span><span class="n">INIT</span><span class="p">:</span> <span class="o">-</span><span class="mi">1</span><span class="o">&gt;</span>,</span><span class="param"> <span class="n">pending_hints</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">output_expression_container</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980141257792&#39;</span><span class="o">&gt;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span>)</span>
<span class="name">DataFrame</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">spark</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657023744704&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="n">expression</span><span class="p">:</span> <span class="n"><a href="../expressions.html#Select">sqlglot.expressions.Select</a></span>,</span><span class="param"> <span class="n">branch_id</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">sequence_id</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">last_op</span><span class="p">:</span> <span class="n">sqlglot</span><span class="o">.</span><span class="n">dataframe</span><span class="o">.</span><span class="n">sql</span><span class="o">.</span><span class="n">operations</span><span class="o">.</span><span class="n">Operation</span> <span class="o">=</span> <span class="o">&lt;</span><span class="n">Operation</span><span class="o">.</span><span class="n">INIT</span><span class="p">:</span> <span class="o">-</span><span class="mi">1</span><span class="o">&gt;</span>,</span><span class="param"> <span class="n">pending_hints</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">output_expression_container</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657023964576&#39;</span><span class="o">&gt;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span>)</span>
<label class="view-source-button" for="DataFrame.__init__-view-source"><span>View Source</span></label>
@ -2096,7 +2096,7 @@
<div class="attr function">
<span class="def">def</span>
<span class="name">sql</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">dialect</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980124350176&#39;</span><span class="o">&gt;</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">optimize</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span><span class="return-annotation">) -> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span>:</span></span>
<span class="name">sql</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">dialect</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014818464&#39;</span><span class="o">&gt;</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">optimize</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span><span class="return-annotation">) -> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span>:</span></span>
<label class="view-source-button" for="DataFrame.sql-view-source"><span>View Source</span></label>
@ -2846,7 +2846,7 @@ is unlikely to come up.</p>
<div class="decorator">@operation(Operation.FROM)</div>
<span class="def">def</span>
<span class="name">fillna</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">value</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980124312224&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="n">subset</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span></span><span class="return-annotation">) -> <span class="n"><a href="#DataFrame">DataFrame</a></span>:</span></span>
<span class="name">fillna</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">value</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657013789248&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="n">subset</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span></span><span class="return-annotation">) -> <span class="n"><a href="#DataFrame">DataFrame</a></span>:</span></span>
<label class="view-source-button" for="DataFrame.fillna-view-source"><span>View Source</span></label>
@ -2915,7 +2915,7 @@ and check if it matches the type of the value provided. If not then make it null
<div class="decorator">@operation(Operation.FROM)</div>
<span class="def">def</span>
<span class="name">replace</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">to_replace</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="nb">float</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">Dict</span><span class="p">]</span>,</span><span class="param"> <span class="n">value</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="nb">float</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">subset</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">Collection</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980124822048&#39;</span><span class="o">&gt;</span><span class="p">],</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980124822048&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span></span><span class="return-annotation">) -> <span class="n"><a href="#DataFrame">DataFrame</a></span>:</span></span>
<span class="name">replace</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">to_replace</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="nb">float</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">Dict</span><span class="p">]</span>,</span><span class="param"> <span class="n">value</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="nb">float</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">subset</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">Collection</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657013976720&#39;</span><span class="o">&gt;</span><span class="p">],</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657013976720&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span></span><span class="return-annotation">) -> <span class="n"><a href="#DataFrame">DataFrame</a></span>:</span></span>
<label class="view-source-button" for="DataFrame.replace-view-source"><span>View Source</span></label>
@ -3120,7 +3120,7 @@ and check if it matches the type of the value provided. If not then make it null
<div class="decorator">@operation(Operation.NO_OP)</div>
<span class="def">def</span>
<span class="name">repartition</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">numPartitions</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980124822048&#39;</span><span class="o">&gt;</span><span class="p">]</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980124822048&#39;</span><span class="o">&gt;</span></span><span class="return-annotation">) -> <span class="n"><a href="#DataFrame">DataFrame</a></span>:</span></span>
<span class="name">repartition</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">numPartitions</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657013976720&#39;</span><span class="o">&gt;</span><span class="p">]</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657013976720&#39;</span><span class="o">&gt;</span></span><span class="return-annotation">) -> <span class="n"><a href="#DataFrame">DataFrame</a></span>:</span></span>
<label class="view-source-button" for="DataFrame.repartition-view-source"><span>View Source</span></label>
@ -3840,7 +3840,7 @@ and check if it matches the type of the value provided. If not then make it null
<input id="Column.__init__-view-source" class="view-source-toggle-state" type="checkbox" aria-hidden="true" tabindex="-1">
<div class="attr function">
<span class="name">Column</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">expression</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span></span>)</span>
<span class="name">Column</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">expression</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span></span>)</span>
<label class="view-source-button" for="Column.__init__-view-source"><span>View Source</span></label>
@ -3884,7 +3884,7 @@ and check if it matches the type of the value provided. If not then make it null
<div class="decorator">@classmethod</div>
<span class="def">def</span>
<span class="name">ensure_col</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="n">value</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<span class="name">ensure_col</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="n">value</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<label class="view-source-button" for="Column.ensure_col-view-source"><span>View Source</span></label>
@ -3905,7 +3905,7 @@ and check if it matches the type of the value provided. If not then make it null
<div class="decorator">@classmethod</div>
<span class="def">def</span>
<span class="name">ensure_cols</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="n">args</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n">List</span><span class="p">[</span><span class="n"><a href="#Column">Column</a></span><span class="p">]</span>:</span></span>
<span class="name">ensure_cols</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="n">args</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n">List</span><span class="p">[</span><span class="n"><a href="#Column">Column</a></span><span class="p">]</span>:</span></span>
<label class="view-source-button" for="Column.ensure_cols-view-source"><span>View Source</span></label>
@ -3926,7 +3926,7 @@ and check if it matches the type of the value provided. If not then make it null
<div class="decorator">@classmethod</div>
<span class="def">def</span>
<span class="name">invoke_anonymous_function</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="n">column</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span><span class="p">]</span>,</span><span class="param"> <span class="n">func_name</span><span class="p">:</span> <span class="nb">str</span>,</span><span class="param"> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span><span class="p">]</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<span class="name">invoke_anonymous_function</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="n">column</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span><span class="p">]</span>,</span><span class="param"> <span class="n">func_name</span><span class="p">:</span> <span class="nb">str</span>,</span><span class="param"> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span><span class="p">]</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<label class="view-source-button" for="Column.invoke_anonymous_function-view-source"><span>View Source</span></label>
@ -3953,7 +3953,7 @@ and check if it matches the type of the value provided. If not then make it null
<div class="decorator">@classmethod</div>
<span class="def">def</span>
<span class="name">invoke_expression_over_column</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="n">column</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span><span class="p">]</span>,</span><span class="param"> <span class="n">callable_expression</span><span class="p">:</span> <span class="n">Callable</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<span class="name">invoke_expression_over_column</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="n">column</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span><span class="p">]</span>,</span><span class="param"> <span class="n">callable_expression</span><span class="p">:</span> <span class="n">Callable</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<label class="view-source-button" for="Column.invoke_expression_over_column-view-source"><span>View Source</span></label>
@ -3992,7 +3992,7 @@ and check if it matches the type of the value provided. If not then make it null
<div class="attr function">
<span class="def">def</span>
<span class="name">binary_op</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">klass</span><span class="p">:</span> <span class="n">Callable</span>,</span><span class="param"> <span class="n">other</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<span class="name">binary_op</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">klass</span><span class="p">:</span> <span class="n">Callable</span>,</span><span class="param"> <span class="n">other</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<label class="view-source-button" for="Column.binary_op-view-source"><span>View Source</span></label>
@ -4013,7 +4013,7 @@ and check if it matches the type of the value provided. If not then make it null
<div class="attr function">
<span class="def">def</span>
<span class="name">inverse_binary_op</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">klass</span><span class="p">:</span> <span class="n">Callable</span>,</span><span class="param"> <span class="n">other</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<span class="name">inverse_binary_op</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">klass</span><span class="p">:</span> <span class="n">Callable</span>,</span><span class="param"> <span class="n">other</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="o">**</span><span class="n">kwargs</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<label class="view-source-button" for="Column.inverse_binary_op-view-source"><span>View Source</span></label>
@ -4607,7 +4607,7 @@ Sqlglot doesn't currently replicate this class so it only accepts a string</p>
<div class="attr function">
<span class="def">def</span>
<span class="name">isin</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">Iterable</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">):</span></span>
<span class="name">isin</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">Iterable</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">):</span></span>
<label class="view-source-button" for="Column.isin-view-source"><span>View Source</span></label>
@ -4628,7 +4628,7 @@ Sqlglot doesn't currently replicate this class so it only accepts a string</p>
<div class="attr function">
<span class="def">def</span>
<span class="name">between</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">lowerBound</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="n">upperBound</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980126740032&#39;</span><span class="o">&gt;</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<span class="name">between</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">lowerBound</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span>,</span><span class="param"> <span class="n">upperBound</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657015887408&#39;</span><span class="o">&gt;</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<label class="view-source-button" for="Column.between-view-source"><span>View Source</span></label>
@ -4663,7 +4663,7 @@ Sqlglot doesn't currently replicate this class so it only accepts a string</p>
<div class="attr function">
<span class="def">def</span>
<span class="name">over</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">window</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980122981280&#39;</span><span class="o">&gt;</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<span class="name">over</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="n">window</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014198032&#39;</span><span class="o">&gt;</span></span><span class="return-annotation">) -> <span class="n"><a href="#Column">Column</a></span>:</span></span>
<label class="view-source-button" for="Column.over-view-source"><span>View Source</span></label>
@ -4908,7 +4908,7 @@ Sqlglot doesn't currently replicate this class so it only accepts a string</p>
<div class="decorator">@classmethod</div>
<span class="def">def</span>
<span class="name">partitionBy</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980123534768&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980123534768&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n"><a href="#WindowSpec">WindowSpec</a></span>:</span></span>
<span class="name">partitionBy</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014627040&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014627040&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n"><a href="#WindowSpec">WindowSpec</a></span>:</span></span>
<label class="view-source-button" for="Window.partitionBy-view-source"><span>View Source</span></label>
@ -4929,7 +4929,7 @@ Sqlglot doesn't currently replicate this class so it only accepts a string</p>
<div class="decorator">@classmethod</div>
<span class="def">def</span>
<span class="name">orderBy</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980123534768&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980123534768&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n"><a href="#WindowSpec">WindowSpec</a></span>:</span></span>
<span class="name">orderBy</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">cls</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014627040&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014627040&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n"><a href="#WindowSpec">WindowSpec</a></span>:</span></span>
<label class="view-source-button" for="Window.orderBy-view-source"><span>View Source</span></label>
@ -5171,7 +5171,7 @@ Sqlglot doesn't currently replicate this class so it only accepts a string</p>
<div class="attr function">
<span class="def">def</span>
<span class="name">partitionBy</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980123534768&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980123534768&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n"><a href="#WindowSpec">WindowSpec</a></span>:</span></span>
<span class="name">partitionBy</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014627040&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014627040&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n"><a href="#WindowSpec">WindowSpec</a></span>:</span></span>
<label class="view-source-button" for="WindowSpec.partitionBy-view-source"><span>View Source</span></label>
@ -5198,7 +5198,7 @@ Sqlglot doesn't currently replicate this class so it only accepts a string</p>
<div class="attr function">
<span class="def">def</span>
<span class="name">orderBy</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980123534768&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;139980123534768&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n"><a href="#WindowSpec">WindowSpec</a></span>:</span></span>
<span class="name">orderBy</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="bp">self</span>,</span><span class="param"> <span class="o">*</span><span class="n">cols</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014627040&#39;</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="o">&lt;</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s1">&#39;140657014627040&#39;</span><span class="o">&gt;</span><span class="p">]]</span></span><span class="return-annotation">) -> <span class="n"><a href="#WindowSpec">WindowSpec</a></span>:</span></span>
<label class="view-source-button" for="WindowSpec.orderBy-view-source"><span>View Source</span></label>

22
docs/sqlglot/dialects/athena.html
View file

File diff suppressed because one or more lines are too long

4363
docs/sqlglot/dialects/bigquery.html
View file

File diff suppressed because one or more lines are too long

5120
docs/sqlglot/dialects/clickhouse.html
View file

File diff suppressed because one or more lines are too long

22
docs/sqlglot/dialects/databricks.html
View file

File diff suppressed because one or more lines are too long

3633
docs/sqlglot/dialects/dialect.html
View file

File diff suppressed because it is too large Load diff

20
docs/sqlglot/dialects/doris.html
View file

File diff suppressed because one or more lines are too long

20
docs/sqlglot/dialects/drill.html
View file

File diff suppressed because one or more lines are too long

3309
docs/sqlglot/dialects/duckdb.html
View file

File diff suppressed because one or more lines are too long

1572
docs/sqlglot/dialects/hive.html
View file

File diff suppressed because one or more lines are too long

4757
docs/sqlglot/dialects/mysql.html
View file

File diff suppressed because one or more lines are too long

24
docs/sqlglot/dialects/oracle.html
View file

File diff suppressed because one or more lines are too long

2203
docs/sqlglot/dialects/postgres.html
View file

File diff suppressed because one or more lines are too long

3141
docs/sqlglot/dialects/presto.html
View file

File diff suppressed because one or more lines are too long

6
docs/sqlglot/dialects/prql.html
View file

File diff suppressed because one or more lines are too long

2230
docs/sqlglot/dialects/redshift.html
View file

File diff suppressed because one or more lines are too long

5429
docs/sqlglot/dialects/snowflake.html
View file

File diff suppressed because one or more lines are too long

301
docs/sqlglot/dialects/spark.html
View file

File diff suppressed because one or more lines are too long

18
docs/sqlglot/dialects/spark2.html
View file

File diff suppressed because one or more lines are too long

22
docs/sqlglot/dialects/sqlite.html
View file

File diff suppressed because one or more lines are too long

20
docs/sqlglot/dialects/starrocks.html
View file

File diff suppressed because one or more lines are too long

18
docs/sqlglot/dialects/tableau.html
View file

File diff suppressed because one or more lines are too long

24
docs/sqlglot/dialects/teradata.html
View file

File diff suppressed because one or more lines are too long

20
docs/sqlglot/dialects/trino.html
View file

File diff suppressed because one or more lines are too long

5449
docs/sqlglot/dialects/tsql.html
View file

File diff suppressed because one or more lines are too long

6
docs/sqlglot/executor/env.html
View file

@ -337,8 +337,10 @@
</span><span id="L-240"><a href="#L-240"><span class="linenos">240</span></a> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">),</span> <span class="mi">2</span><span class="p">)</span>
</span><span id="L-241"><a href="#L-241"><span class="linenos">241</span></a> <span class="k">if</span> <span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">args</span><span class="p">[</span><span class="n">x</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">)</span>
</span><span id="L-242"><a href="#L-242"><span class="linenos">242</span></a> <span class="p">},</span>
</span><span id="L-243"><a href="#L-243"><span class="linenos">243</span></a> <span class="s2">&quot;UNIXTOTIME&quot;</span><span class="p">:</span> <span class="n">null_if_any</span><span class="p">(</span><span class="k">lambda</span> <span class="n">arg</span><span class="p">:</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcfromtimestamp</span><span class="p">(</span><span class="n">arg</span><span class="p">)),</span>
</span><span id="L-244"><a href="#L-244"><span class="linenos">244</span></a><span class="p">}</span>
</span><span id="L-243"><a href="#L-243"><span class="linenos">243</span></a> <span class="s2">&quot;UNIXTOTIME&quot;</span><span class="p">:</span> <span class="n">null_if_any</span><span class="p">(</span>
</span><span id="L-244"><a href="#L-244"><span class="linenos">244</span></a> <span class="k">lambda</span> <span class="n">arg</span><span class="p">:</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">fromtimestamp</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span>
</span><span id="L-245"><a href="#L-245"><span class="linenos">245</span></a> <span class="p">),</span>
</span><span id="L-246"><a href="#L-246"><span class="linenos">246</span></a><span class="p">}</span>
</span></pre></div>

11
docs/sqlglot/executor/python.html
View file

File diff suppressed because one or more lines are too long

24201
docs/sqlglot/expressions.html
View file

File diff suppressed because one or more lines are too long

20586
docs/sqlglot/generator.html
View file

File diff suppressed because one or more lines are too long

2
docs/sqlglot/helper.html
View file

@ -1893,7 +1893,7 @@ belong to some totally-ordered set.</p>
<section id="DATE_UNITS">
<div class="attr variable">
<span class="name">DATE_UNITS</span> =
<span class="default_value">{&#39;quarter&#39;, &#39;year_month&#39;, &#39;week&#39;, &#39;year&#39;, &#39;day&#39;, &#39;month&#39;}</span>
<span class="default_value">{&#39;year_month&#39;, &#39;week&#39;, &#39;year&#39;, &#39;month&#39;, &#39;day&#39;, &#39;quarter&#39;}</span>
</div>

2
docs/sqlglot/jsonpath.html
View file

@ -577,7 +577,7 @@
<div class="attr variable">
<span class="name">ALL_JSON_PATH_PARTS</span> =
<input id="ALL_JSON_PATH_PARTS-view-value" class="view-value-toggle-state" type="checkbox" aria-hidden="true" tabindex="-1">
<label class="view-value-button pdoc-button" for="ALL_JSON_PATH_PARTS-view-value"></label><span class="default_value">{&lt;class &#39;<a href="expressions.html#JSONPathRecursive">sqlglot.expressions.JSONPathRecursive</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathWildcard">sqlglot.expressions.JSONPathWildcard</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathUnion">sqlglot.expressions.JSONPathUnion</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathKey">sqlglot.expressions.JSONPathKey</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSubscript">sqlglot.expressions.JSONPathSubscript</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathFilter">sqlglot.expressions.JSONPathFilter</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSelector">sqlglot.expressions.JSONPathSelector</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSlice">sqlglot.expressions.JSONPathSlice</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathScript">sqlglot.expressions.JSONPathScript</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathRoot">sqlglot.expressions.JSONPathRoot</a>&#39;&gt;}</span>
<label class="view-value-button pdoc-button" for="ALL_JSON_PATH_PARTS-view-value"></label><span class="default_value">{&lt;class &#39;<a href="expressions.html#JSONPathFilter">sqlglot.expressions.JSONPathFilter</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathUnion">sqlglot.expressions.JSONPathUnion</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSubscript">sqlglot.expressions.JSONPathSubscript</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSelector">sqlglot.expressions.JSONPathSelector</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathWildcard">sqlglot.expressions.JSONPathWildcard</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSlice">sqlglot.expressions.JSONPathSlice</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathScript">sqlglot.expressions.JSONPathScript</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathRoot">sqlglot.expressions.JSONPathRoot</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathRecursive">sqlglot.expressions.JSONPathRecursive</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathKey">sqlglot.expressions.JSONPathKey</a>&#39;&gt;}</span>
</div>

590
docs/sqlglot/lineage.html
View file

@ -347,136 +347,140 @@
</span><span id="L-229"><a href="#L-229"><span class="linenos">229</span></a> <span class="k">for</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
</span><span id="L-230"><a href="#L-230"><span class="linenos">230</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">):</span>
</span><span id="L-231"><a href="#L-231"><span class="linenos">231</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">source</span><span class="o">.</span><span class="n">expression</span>
</span><span id="L-232"><a href="#L-232"><span class="linenos">232</span></a> <span class="n">node</span><span class="o">.</span><span class="n">downstream</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">select</span><span class="o">.</span><span class="n">sql</span><span class="p">(),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">source</span><span class="p">))</span>
</span><span id="L-233"><a href="#L-233"><span class="linenos">233</span></a>
</span><span id="L-234"><a href="#L-234"><span class="linenos">234</span></a> <span class="c1"># Find all columns that went into creating this one to list their lineage nodes.</span>
</span><span id="L-235"><a href="#L-235"><span class="linenos">235</span></a> <span class="n">source_columns</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">find_all_in_scope</span><span class="p">(</span><span class="n">select</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Column</span><span class="p">))</span>
</span><span id="L-236"><a href="#L-236"><span class="linenos">236</span></a>
</span><span id="L-237"><a href="#L-237"><span class="linenos">237</span></a> <span class="c1"># If the source is a UDTF find columns used in the UTDF to generate the table</span>
</span><span id="L-238"><a href="#L-238"><span class="linenos">238</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">):</span>
</span><span id="L-239"><a href="#L-239"><span class="linenos">239</span></a> <span class="n">source_columns</span> <span class="o">|=</span> <span class="nb">set</span><span class="p">(</span><span class="n">source</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Column</span><span class="p">))</span>
</span><span id="L-240"><a href="#L-240"><span class="linenos">240</span></a> <span class="n">derived_tables</span> <span class="o">=</span> <span class="p">[</span>
</span><span id="L-241"><a href="#L-241"><span class="linenos">241</span></a> <span class="n">source</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-242"><a href="#L-242"><span class="linenos">242</span></a> <span class="k">for</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
</span><span id="L-243"><a href="#L-243"><span class="linenos">243</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">)</span> <span class="ow">and</span> <span class="n">source</span><span class="o">.</span><span class="n">is_derived_table</span>
</span><span id="L-244"><a href="#L-244"><span class="linenos">244</span></a> <span class="p">]</span>
</span><span id="L-245"><a href="#L-245"><span class="linenos">245</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-246"><a href="#L-246"><span class="linenos">246</span></a> <span class="n">derived_tables</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">derived_tables</span>
</span><span id="L-247"><a href="#L-247"><span class="linenos">247</span></a>
</span><span id="L-248"><a href="#L-248"><span class="linenos">248</span></a> <span class="n">source_names</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="L-249"><a href="#L-249"><span class="linenos">249</span></a> <span class="n">dt</span><span class="o">.</span><span class="n">alias</span><span class="p">:</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span>
</span><span id="L-250"><a href="#L-250"><span class="linenos">250</span></a> <span class="k">for</span> <span class="n">dt</span> <span class="ow">in</span> <span class="n">derived_tables</span>
</span><span id="L-251"><a href="#L-251"><span class="linenos">251</span></a> <span class="k">if</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span> <span class="ow">and</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;source: &quot;</span><span class="p">)</span>
</span><span id="L-252"><a href="#L-252"><span class="linenos">252</span></a> <span class="p">}</span>
</span><span id="L-253"><a href="#L-253"><span class="linenos">253</span></a>
</span><span id="L-254"><a href="#L-254"><span class="linenos">254</span></a> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">source_columns</span><span class="p">:</span>
</span><span id="L-255"><a href="#L-255"><span class="linenos">255</span></a> <span class="n">table</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="n">table</span>
</span><span id="L-256"><a href="#L-256"><span class="linenos">256</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">)</span>
</span><span id="L-257"><a href="#L-257"><span class="linenos">257</span></a>
</span><span id="L-258"><a href="#L-258"><span class="linenos">258</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">):</span>
</span><span id="L-259"><a href="#L-259"><span class="linenos">259</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-260"><a href="#L-260"><span class="linenos">260</span></a> <span class="k">if</span> <span class="n">source</span><span class="o">.</span><span class="n">scope_type</span> <span class="o">==</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">DERIVED_TABLE</span> <span class="ow">and</span> <span class="n">table</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">source_names</span><span class="p">:</span>
</span><span id="L-261"><a href="#L-261"><span class="linenos">261</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="n">table</span>
</span><span id="L-262"><a href="#L-262"><span class="linenos">262</span></a> <span class="k">elif</span> <span class="n">source</span><span class="o">.</span><span class="n">scope_type</span> <span class="o">==</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">CTE</span><span class="p">:</span>
</span><span id="L-263"><a href="#L-263"><span class="linenos">263</span></a> <span class="n">selected_node</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">selected_sources</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">))</span>
</span><span id="L-264"><a href="#L-264"><span class="linenos">264</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="n">selected_node</span><span class="o">.</span><span class="n">name</span> <span class="k">if</span> <span class="n">selected_node</span> <span class="k">else</span> <span class="kc">None</span>
</span><span id="L-265"><a href="#L-265"><span class="linenos">265</span></a> <span class="c1"># The table itself came from a more specific scope. Recurse into that one using the unaliased column name.</span>
</span><span id="L-266"><a href="#L-266"><span class="linenos">266</span></a> <span class="n">to_node</span><span class="p">(</span>
</span><span id="L-267"><a href="#L-267"><span class="linenos">267</span></a> <span class="n">c</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
</span><span id="L-268"><a href="#L-268"><span class="linenos">268</span></a> <span class="n">scope</span><span class="o">=</span><span class="n">source</span><span class="p">,</span>
</span><span id="L-269"><a href="#L-269"><span class="linenos">269</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="L-270"><a href="#L-270"><span class="linenos">270</span></a> <span class="n">scope_name</span><span class="o">=</span><span class="n">table</span><span class="p">,</span>
</span><span id="L-271"><a href="#L-271"><span class="linenos">271</span></a> <span class="n">upstream</span><span class="o">=</span><span class="n">node</span><span class="p">,</span>
</span><span id="L-272"><a href="#L-272"><span class="linenos">272</span></a> <span class="n">source_name</span><span class="o">=</span><span class="n">source_names</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">)</span> <span class="ow">or</span> <span class="n">source_name</span><span class="p">,</span>
</span><span id="L-273"><a href="#L-273"><span class="linenos">273</span></a> <span class="n">reference_node_name</span><span class="o">=</span><span class="n">reference_node_name</span><span class="p">,</span>
</span><span id="L-274"><a href="#L-274"><span class="linenos">274</span></a> <span class="n">trim_selects</span><span class="o">=</span><span class="n">trim_selects</span><span class="p">,</span>
</span><span id="L-275"><a href="#L-275"><span class="linenos">275</span></a> <span class="p">)</span>
</span><span id="L-276"><a href="#L-276"><span class="linenos">276</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-277"><a href="#L-277"><span class="linenos">277</span></a> <span class="c1"># The source is not a scope - we&#39;ve reached the end of the line. At this point, if a source is not found</span>
</span><span id="L-278"><a href="#L-278"><span class="linenos">278</span></a> <span class="c1"># it means this column&#39;s lineage is unknown. This can happen if the definition of a source used in a query</span>
</span><span id="L-279"><a href="#L-279"><span class="linenos">279</span></a> <span class="c1"># is not passed into the `sources` map.</span>
</span><span id="L-280"><a href="#L-280"><span class="linenos">280</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">source</span> <span class="ow">or</span> <span class="n">exp</span><span class="o">.</span><span class="n">Placeholder</span><span class="p">()</span>
</span><span id="L-281"><a href="#L-281"><span class="linenos">281</span></a> <span class="n">node</span><span class="o">.</span><span class="n">downstream</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">c</span><span class="o">.</span><span class="n">sql</span><span class="p">(),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">source</span><span class="p">))</span>
</span><span id="L-282"><a href="#L-282"><span class="linenos">282</span></a>
</span><span id="L-283"><a href="#L-283"><span class="linenos">283</span></a> <span class="k">return</span> <span class="n">node</span>
</span><span id="L-284"><a href="#L-284"><span class="linenos">284</span></a>
</span><span id="L-285"><a href="#L-285"><span class="linenos">285</span></a>
</span><span id="L-286"><a href="#L-286"><span class="linenos">286</span></a><span class="k">class</span> <span class="nc">GraphHTML</span><span class="p">:</span>
</span><span id="L-287"><a href="#L-287"><span class="linenos">287</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;Node to HTML generator using vis.js.</span>
</span><span id="L-232"><a href="#L-232"><span class="linenos">232</span></a> <span class="n">node</span><span class="o">.</span><span class="n">downstream</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
</span><span id="L-233"><a href="#L-233"><span class="linenos">233</span></a> <span class="n">Node</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">select</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="n">comments</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">source</span><span class="p">)</span>
</span><span id="L-234"><a href="#L-234"><span class="linenos">234</span></a> <span class="p">)</span>
</span><span id="L-235"><a href="#L-235"><span class="linenos">235</span></a>
</span><span id="L-236"><a href="#L-236"><span class="linenos">236</span></a> <span class="c1"># Find all columns that went into creating this one to list their lineage nodes.</span>
</span><span id="L-237"><a href="#L-237"><span class="linenos">237</span></a> <span class="n">source_columns</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">find_all_in_scope</span><span class="p">(</span><span class="n">select</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Column</span><span class="p">))</span>
</span><span id="L-238"><a href="#L-238"><span class="linenos">238</span></a>
</span><span id="L-239"><a href="#L-239"><span class="linenos">239</span></a> <span class="c1"># If the source is a UDTF find columns used in the UTDF to generate the table</span>
</span><span id="L-240"><a href="#L-240"><span class="linenos">240</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">):</span>
</span><span id="L-241"><a href="#L-241"><span class="linenos">241</span></a> <span class="n">source_columns</span> <span class="o">|=</span> <span class="nb">set</span><span class="p">(</span><span class="n">source</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Column</span><span class="p">))</span>
</span><span id="L-242"><a href="#L-242"><span class="linenos">242</span></a> <span class="n">derived_tables</span> <span class="o">=</span> <span class="p">[</span>
</span><span id="L-243"><a href="#L-243"><span class="linenos">243</span></a> <span class="n">source</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-244"><a href="#L-244"><span class="linenos">244</span></a> <span class="k">for</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
</span><span id="L-245"><a href="#L-245"><span class="linenos">245</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">)</span> <span class="ow">and</span> <span class="n">source</span><span class="o">.</span><span class="n">is_derived_table</span>
</span><span id="L-246"><a href="#L-246"><span class="linenos">246</span></a> <span class="p">]</span>
</span><span id="L-247"><a href="#L-247"><span class="linenos">247</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-248"><a href="#L-248"><span class="linenos">248</span></a> <span class="n">derived_tables</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">derived_tables</span>
</span><span id="L-249"><a href="#L-249"><span class="linenos">249</span></a>
</span><span id="L-250"><a href="#L-250"><span class="linenos">250</span></a> <span class="n">source_names</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="L-251"><a href="#L-251"><span class="linenos">251</span></a> <span class="n">dt</span><span class="o">.</span><span class="n">alias</span><span class="p">:</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span>
</span><span id="L-252"><a href="#L-252"><span class="linenos">252</span></a> <span class="k">for</span> <span class="n">dt</span> <span class="ow">in</span> <span class="n">derived_tables</span>
</span><span id="L-253"><a href="#L-253"><span class="linenos">253</span></a> <span class="k">if</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span> <span class="ow">and</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;source: &quot;</span><span class="p">)</span>
</span><span id="L-254"><a href="#L-254"><span class="linenos">254</span></a> <span class="p">}</span>
</span><span id="L-255"><a href="#L-255"><span class="linenos">255</span></a>
</span><span id="L-256"><a href="#L-256"><span class="linenos">256</span></a> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">source_columns</span><span class="p">:</span>
</span><span id="L-257"><a href="#L-257"><span class="linenos">257</span></a> <span class="n">table</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="n">table</span>
</span><span id="L-258"><a href="#L-258"><span class="linenos">258</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">)</span>
</span><span id="L-259"><a href="#L-259"><span class="linenos">259</span></a>
</span><span id="L-260"><a href="#L-260"><span class="linenos">260</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">):</span>
</span><span id="L-261"><a href="#L-261"><span class="linenos">261</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-262"><a href="#L-262"><span class="linenos">262</span></a> <span class="k">if</span> <span class="n">source</span><span class="o">.</span><span class="n">scope_type</span> <span class="o">==</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">DERIVED_TABLE</span> <span class="ow">and</span> <span class="n">table</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">source_names</span><span class="p">:</span>
</span><span id="L-263"><a href="#L-263"><span class="linenos">263</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="n">table</span>
</span><span id="L-264"><a href="#L-264"><span class="linenos">264</span></a> <span class="k">elif</span> <span class="n">source</span><span class="o">.</span><span class="n">scope_type</span> <span class="o">==</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">CTE</span><span class="p">:</span>
</span><span id="L-265"><a href="#L-265"><span class="linenos">265</span></a> <span class="n">selected_node</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">selected_sources</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">))</span>
</span><span id="L-266"><a href="#L-266"><span class="linenos">266</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="n">selected_node</span><span class="o">.</span><span class="n">name</span> <span class="k">if</span> <span class="n">selected_node</span> <span class="k">else</span> <span class="kc">None</span>
</span><span id="L-267"><a href="#L-267"><span class="linenos">267</span></a> <span class="c1"># The table itself came from a more specific scope. Recurse into that one using the unaliased column name.</span>
</span><span id="L-268"><a href="#L-268"><span class="linenos">268</span></a> <span class="n">to_node</span><span class="p">(</span>
</span><span id="L-269"><a href="#L-269"><span class="linenos">269</span></a> <span class="n">c</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
</span><span id="L-270"><a href="#L-270"><span class="linenos">270</span></a> <span class="n">scope</span><span class="o">=</span><span class="n">source</span><span class="p">,</span>
</span><span id="L-271"><a href="#L-271"><span class="linenos">271</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="L-272"><a href="#L-272"><span class="linenos">272</span></a> <span class="n">scope_name</span><span class="o">=</span><span class="n">table</span><span class="p">,</span>
</span><span id="L-273"><a href="#L-273"><span class="linenos">273</span></a> <span class="n">upstream</span><span class="o">=</span><span class="n">node</span><span class="p">,</span>
</span><span id="L-274"><a href="#L-274"><span class="linenos">274</span></a> <span class="n">source_name</span><span class="o">=</span><span class="n">source_names</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">)</span> <span class="ow">or</span> <span class="n">source_name</span><span class="p">,</span>
</span><span id="L-275"><a href="#L-275"><span class="linenos">275</span></a> <span class="n">reference_node_name</span><span class="o">=</span><span class="n">reference_node_name</span><span class="p">,</span>
</span><span id="L-276"><a href="#L-276"><span class="linenos">276</span></a> <span class="n">trim_selects</span><span class="o">=</span><span class="n">trim_selects</span><span class="p">,</span>
</span><span id="L-277"><a href="#L-277"><span class="linenos">277</span></a> <span class="p">)</span>
</span><span id="L-278"><a href="#L-278"><span class="linenos">278</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-279"><a href="#L-279"><span class="linenos">279</span></a> <span class="c1"># The source is not a scope - we&#39;ve reached the end of the line. At this point, if a source is not found</span>
</span><span id="L-280"><a href="#L-280"><span class="linenos">280</span></a> <span class="c1"># it means this column&#39;s lineage is unknown. This can happen if the definition of a source used in a query</span>
</span><span id="L-281"><a href="#L-281"><span class="linenos">281</span></a> <span class="c1"># is not passed into the `sources` map.</span>
</span><span id="L-282"><a href="#L-282"><span class="linenos">282</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">source</span> <span class="ow">or</span> <span class="n">exp</span><span class="o">.</span><span class="n">Placeholder</span><span class="p">()</span>
</span><span id="L-283"><a href="#L-283"><span class="linenos">283</span></a> <span class="n">node</span><span class="o">.</span><span class="n">downstream</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
</span><span id="L-284"><a href="#L-284"><span class="linenos">284</span></a> <span class="n">Node</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">c</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="n">comments</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">source</span><span class="p">)</span>
</span><span id="L-285"><a href="#L-285"><span class="linenos">285</span></a> <span class="p">)</span>
</span><span id="L-286"><a href="#L-286"><span class="linenos">286</span></a>
</span><span id="L-287"><a href="#L-287"><span class="linenos">287</span></a> <span class="k">return</span> <span class="n">node</span>
</span><span id="L-288"><a href="#L-288"><span class="linenos">288</span></a>
</span><span id="L-289"><a href="#L-289"><span class="linenos">289</span></a><span class="sd"> https://visjs.github.io/vis-network/docs/network/</span>
</span><span id="L-290"><a href="#L-290"><span class="linenos">290</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-291"><a href="#L-291"><span class="linenos">291</span></a>
</span><span id="L-292"><a href="#L-292"><span class="linenos">292</span></a> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span>
</span><span id="L-293"><a href="#L-293"><span class="linenos">293</span></a> <span class="bp">self</span><span class="p">,</span> <span class="n">nodes</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">edges</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">,</span> <span class="n">imports</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span> <span class="n">options</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-294"><a href="#L-294"><span class="linenos">294</span></a> <span class="p">):</span>
</span><span id="L-295"><a href="#L-295"><span class="linenos">295</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span> <span class="o">=</span> <span class="n">imports</span>
</span><span id="L-296"><a href="#L-296"><span class="linenos">296</span></a>
</span><span id="L-297"><a href="#L-297"><span class="linenos">297</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="L-298"><a href="#L-298"><span class="linenos">298</span></a> <span class="s2">&quot;height&quot;</span><span class="p">:</span> <span class="s2">&quot;500px&quot;</span><span class="p">,</span>
</span><span id="L-299"><a href="#L-299"><span class="linenos">299</span></a> <span class="s2">&quot;width&quot;</span><span class="p">:</span> <span class="s2">&quot;100%&quot;</span><span class="p">,</span>
</span><span id="L-300"><a href="#L-300"><span class="linenos">300</span></a> <span class="s2">&quot;layout&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-301"><a href="#L-301"><span class="linenos">301</span></a> <span class="s2">&quot;hierarchical&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-302"><a href="#L-302"><span class="linenos">302</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-303"><a href="#L-303"><span class="linenos">303</span></a> <span class="s2">&quot;nodeSpacing&quot;</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
</span><span id="L-304"><a href="#L-304"><span class="linenos">304</span></a> <span class="s2">&quot;sortMethod&quot;</span><span class="p">:</span> <span class="s2">&quot;directed&quot;</span><span class="p">,</span>
</span><span id="L-305"><a href="#L-305"><span class="linenos">305</span></a> <span class="p">},</span>
</span><span id="L-306"><a href="#L-306"><span class="linenos">306</span></a> <span class="p">},</span>
</span><span id="L-307"><a href="#L-307"><span class="linenos">307</span></a> <span class="s2">&quot;interaction&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-308"><a href="#L-308"><span class="linenos">308</span></a> <span class="s2">&quot;dragNodes&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-309"><a href="#L-309"><span class="linenos">309</span></a> <span class="s2">&quot;selectable&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-289"><a href="#L-289"><span class="linenos">289</span></a>
</span><span id="L-290"><a href="#L-290"><span class="linenos">290</span></a><span class="k">class</span> <span class="nc">GraphHTML</span><span class="p">:</span>
</span><span id="L-291"><a href="#L-291"><span class="linenos">291</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;Node to HTML generator using vis.js.</span>
</span><span id="L-292"><a href="#L-292"><span class="linenos">292</span></a>
</span><span id="L-293"><a href="#L-293"><span class="linenos">293</span></a><span class="sd"> https://visjs.github.io/vis-network/docs/network/</span>
</span><span id="L-294"><a href="#L-294"><span class="linenos">294</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-295"><a href="#L-295"><span class="linenos">295</span></a>
</span><span id="L-296"><a href="#L-296"><span class="linenos">296</span></a> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span>
</span><span id="L-297"><a href="#L-297"><span class="linenos">297</span></a> <span class="bp">self</span><span class="p">,</span> <span class="n">nodes</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">edges</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">,</span> <span class="n">imports</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span> <span class="n">options</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-298"><a href="#L-298"><span class="linenos">298</span></a> <span class="p">):</span>
</span><span id="L-299"><a href="#L-299"><span class="linenos">299</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span> <span class="o">=</span> <span class="n">imports</span>
</span><span id="L-300"><a href="#L-300"><span class="linenos">300</span></a>
</span><span id="L-301"><a href="#L-301"><span class="linenos">301</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="L-302"><a href="#L-302"><span class="linenos">302</span></a> <span class="s2">&quot;height&quot;</span><span class="p">:</span> <span class="s2">&quot;500px&quot;</span><span class="p">,</span>
</span><span id="L-303"><a href="#L-303"><span class="linenos">303</span></a> <span class="s2">&quot;width&quot;</span><span class="p">:</span> <span class="s2">&quot;100%&quot;</span><span class="p">,</span>
</span><span id="L-304"><a href="#L-304"><span class="linenos">304</span></a> <span class="s2">&quot;layout&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-305"><a href="#L-305"><span class="linenos">305</span></a> <span class="s2">&quot;hierarchical&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-306"><a href="#L-306"><span class="linenos">306</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-307"><a href="#L-307"><span class="linenos">307</span></a> <span class="s2">&quot;nodeSpacing&quot;</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
</span><span id="L-308"><a href="#L-308"><span class="linenos">308</span></a> <span class="s2">&quot;sortMethod&quot;</span><span class="p">:</span> <span class="s2">&quot;directed&quot;</span><span class="p">,</span>
</span><span id="L-309"><a href="#L-309"><span class="linenos">309</span></a> <span class="p">},</span>
</span><span id="L-310"><a href="#L-310"><span class="linenos">310</span></a> <span class="p">},</span>
</span><span id="L-311"><a href="#L-311"><span class="linenos">311</span></a> <span class="s2">&quot;physics&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-312"><a href="#L-312"><span class="linenos">312</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-313"><a href="#L-313"><span class="linenos">313</span></a> <span class="p">},</span>
</span><span id="L-314"><a href="#L-314"><span class="linenos">314</span></a> <span class="s2">&quot;edges&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-315"><a href="#L-315"><span class="linenos">315</span></a> <span class="s2">&quot;arrows&quot;</span><span class="p">:</span> <span class="s2">&quot;to&quot;</span><span class="p">,</span>
</span><span id="L-316"><a href="#L-316"><span class="linenos">316</span></a> <span class="p">},</span>
</span><span id="L-317"><a href="#L-317"><span class="linenos">317</span></a> <span class="s2">&quot;nodes&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-318"><a href="#L-318"><span class="linenos">318</span></a> <span class="s2">&quot;font&quot;</span><span class="p">:</span> <span class="s2">&quot;20px monaco&quot;</span><span class="p">,</span>
</span><span id="L-319"><a href="#L-319"><span class="linenos">319</span></a> <span class="s2">&quot;shape&quot;</span><span class="p">:</span> <span class="s2">&quot;box&quot;</span><span class="p">,</span>
</span><span id="L-320"><a href="#L-320"><span class="linenos">320</span></a> <span class="s2">&quot;widthConstraint&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-321"><a href="#L-321"><span class="linenos">321</span></a> <span class="s2">&quot;maximum&quot;</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
</span><span id="L-322"><a href="#L-322"><span class="linenos">322</span></a> <span class="p">},</span>
</span><span id="L-323"><a href="#L-323"><span class="linenos">323</span></a> <span class="p">},</span>
</span><span id="L-324"><a href="#L-324"><span class="linenos">324</span></a> <span class="o">**</span><span class="p">(</span><span class="n">options</span> <span class="ow">or</span> <span class="p">{}),</span>
</span><span id="L-325"><a href="#L-325"><span class="linenos">325</span></a> <span class="p">}</span>
</span><span id="L-326"><a href="#L-326"><span class="linenos">326</span></a>
</span><span id="L-327"><a href="#L-327"><span class="linenos">327</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">nodes</span> <span class="o">=</span> <span class="n">nodes</span>
</span><span id="L-328"><a href="#L-328"><span class="linenos">328</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">edges</span> <span class="o">=</span> <span class="n">edges</span>
</span><span id="L-329"><a href="#L-329"><span class="linenos">329</span></a>
</span><span id="L-330"><a href="#L-330"><span class="linenos">330</span></a> <span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span><span id="L-331"><a href="#L-331"><span class="linenos">331</span></a> <span class="n">nodes</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">nodes</span><span class="o">.</span><span class="n">values</span><span class="p">()))</span>
</span><span id="L-332"><a href="#L-332"><span class="linenos">332</span></a> <span class="n">edges</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">edges</span><span class="p">)</span>
</span><span id="L-333"><a href="#L-333"><span class="linenos">333</span></a> <span class="n">options</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">options</span><span class="p">)</span>
</span><span id="L-334"><a href="#L-334"><span class="linenos">334</span></a> <span class="n">imports</span> <span class="o">=</span> <span class="p">(</span>
</span><span id="L-335"><a href="#L-335"><span class="linenos">335</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;&lt;script type=&quot;text/javascript&quot; src=&quot;https://unpkg.com/vis-data@latest/peer/umd/vis-data.min.js&quot;&gt;&lt;/script&gt;</span>
</span><span id="L-336"><a href="#L-336"><span class="linenos">336</span></a><span class="sd"> &lt;script type=&quot;text/javascript&quot; src=&quot;https://unpkg.com/vis-network@latest/peer/umd/vis-network.min.js&quot;&gt;&lt;/script&gt;</span>
</span><span id="L-337"><a href="#L-337"><span class="linenos">337</span></a><span class="sd"> &lt;link rel=&quot;stylesheet&quot; type=&quot;text/css&quot; href=&quot;https://unpkg.com/vis-network/styles/vis-network.min.css&quot; /&gt;&quot;&quot;&quot;</span>
</span><span id="L-338"><a href="#L-338"><span class="linenos">338</span></a> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span>
</span><span id="L-339"><a href="#L-339"><span class="linenos">339</span></a> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
</span><span id="L-340"><a href="#L-340"><span class="linenos">340</span></a> <span class="p">)</span>
</span><span id="L-341"><a href="#L-341"><span class="linenos">341</span></a>
</span><span id="L-342"><a href="#L-342"><span class="linenos">342</span></a> <span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;&quot;&quot;&lt;div&gt;</span>
</span><span id="L-343"><a href="#L-343"><span class="linenos">343</span></a><span class="s2"> &lt;div id=&quot;sqlglot-lineage&quot;&gt;&lt;/div&gt;</span>
</span><span id="L-344"><a href="#L-344"><span class="linenos">344</span></a><span class="s2"> </span><span class="si">{</span><span class="n">imports</span><span class="si">}</span>
</span><span id="L-345"><a href="#L-345"><span class="linenos">345</span></a><span class="s2"> &lt;script type=&quot;text/javascript&quot;&gt;</span>
</span><span id="L-346"><a href="#L-346"><span class="linenos">346</span></a><span class="s2"> var nodes = new vis.DataSet(</span><span class="si">{</span><span class="n">nodes</span><span class="si">}</span><span class="s2">)</span>
</span><span id="L-347"><a href="#L-347"><span class="linenos">347</span></a><span class="s2"> nodes.forEach(row =&gt; row[&quot;title&quot;] = new DOMParser().parseFromString(row[&quot;title&quot;], &quot;text/html&quot;).body.childNodes[0])</span>
</span><span id="L-348"><a href="#L-348"><span class="linenos">348</span></a>
</span><span id="L-349"><a href="#L-349"><span class="linenos">349</span></a><span class="s2"> new vis.Network(</span>
</span><span id="L-350"><a href="#L-350"><span class="linenos">350</span></a><span class="s2"> document.getElementById(&quot;sqlglot-lineage&quot;),</span>
</span><span id="L-351"><a href="#L-351"><span class="linenos">351</span></a><span class="s2"> </span><span class="se">{{</span>
</span><span id="L-352"><a href="#L-352"><span class="linenos">352</span></a><span class="s2"> nodes: nodes,</span>
</span><span id="L-353"><a href="#L-353"><span class="linenos">353</span></a><span class="s2"> edges: new vis.DataSet(</span><span class="si">{</span><span class="n">edges</span><span class="si">}</span><span class="s2">)</span>
</span><span id="L-354"><a href="#L-354"><span class="linenos">354</span></a><span class="s2"> </span><span class="se">}}</span><span class="s2">,</span>
</span><span id="L-355"><a href="#L-355"><span class="linenos">355</span></a><span class="s2"> </span><span class="si">{</span><span class="n">options</span><span class="si">}</span><span class="s2">,</span>
</span><span id="L-356"><a href="#L-356"><span class="linenos">356</span></a><span class="s2"> )</span>
</span><span id="L-357"><a href="#L-357"><span class="linenos">357</span></a><span class="s2"> &lt;/script&gt;</span>
</span><span id="L-358"><a href="#L-358"><span class="linenos">358</span></a><span class="s2">&lt;/div&gt;&quot;&quot;&quot;</span>
</span><span id="L-359"><a href="#L-359"><span class="linenos">359</span></a>
</span><span id="L-360"><a href="#L-360"><span class="linenos">360</span></a> <span class="k">def</span> <span class="nf">_repr_html_</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span><span id="L-361"><a href="#L-361"><span class="linenos">361</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span>
</span><span id="L-311"><a href="#L-311"><span class="linenos">311</span></a> <span class="s2">&quot;interaction&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-312"><a href="#L-312"><span class="linenos">312</span></a> <span class="s2">&quot;dragNodes&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-313"><a href="#L-313"><span class="linenos">313</span></a> <span class="s2">&quot;selectable&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-314"><a href="#L-314"><span class="linenos">314</span></a> <span class="p">},</span>
</span><span id="L-315"><a href="#L-315"><span class="linenos">315</span></a> <span class="s2">&quot;physics&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-316"><a href="#L-316"><span class="linenos">316</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-317"><a href="#L-317"><span class="linenos">317</span></a> <span class="p">},</span>
</span><span id="L-318"><a href="#L-318"><span class="linenos">318</span></a> <span class="s2">&quot;edges&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-319"><a href="#L-319"><span class="linenos">319</span></a> <span class="s2">&quot;arrows&quot;</span><span class="p">:</span> <span class="s2">&quot;to&quot;</span><span class="p">,</span>
</span><span id="L-320"><a href="#L-320"><span class="linenos">320</span></a> <span class="p">},</span>
</span><span id="L-321"><a href="#L-321"><span class="linenos">321</span></a> <span class="s2">&quot;nodes&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-322"><a href="#L-322"><span class="linenos">322</span></a> <span class="s2">&quot;font&quot;</span><span class="p">:</span> <span class="s2">&quot;20px monaco&quot;</span><span class="p">,</span>
</span><span id="L-323"><a href="#L-323"><span class="linenos">323</span></a> <span class="s2">&quot;shape&quot;</span><span class="p">:</span> <span class="s2">&quot;box&quot;</span><span class="p">,</span>
</span><span id="L-324"><a href="#L-324"><span class="linenos">324</span></a> <span class="s2">&quot;widthConstraint&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="L-325"><a href="#L-325"><span class="linenos">325</span></a> <span class="s2">&quot;maximum&quot;</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
</span><span id="L-326"><a href="#L-326"><span class="linenos">326</span></a> <span class="p">},</span>
</span><span id="L-327"><a href="#L-327"><span class="linenos">327</span></a> <span class="p">},</span>
</span><span id="L-328"><a href="#L-328"><span class="linenos">328</span></a> <span class="o">**</span><span class="p">(</span><span class="n">options</span> <span class="ow">or</span> <span class="p">{}),</span>
</span><span id="L-329"><a href="#L-329"><span class="linenos">329</span></a> <span class="p">}</span>
</span><span id="L-330"><a href="#L-330"><span class="linenos">330</span></a>
</span><span id="L-331"><a href="#L-331"><span class="linenos">331</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">nodes</span> <span class="o">=</span> <span class="n">nodes</span>
</span><span id="L-332"><a href="#L-332"><span class="linenos">332</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">edges</span> <span class="o">=</span> <span class="n">edges</span>
</span><span id="L-333"><a href="#L-333"><span class="linenos">333</span></a>
</span><span id="L-334"><a href="#L-334"><span class="linenos">334</span></a> <span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span><span id="L-335"><a href="#L-335"><span class="linenos">335</span></a> <span class="n">nodes</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">nodes</span><span class="o">.</span><span class="n">values</span><span class="p">()))</span>
</span><span id="L-336"><a href="#L-336"><span class="linenos">336</span></a> <span class="n">edges</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">edges</span><span class="p">)</span>
</span><span id="L-337"><a href="#L-337"><span class="linenos">337</span></a> <span class="n">options</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">options</span><span class="p">)</span>
</span><span id="L-338"><a href="#L-338"><span class="linenos">338</span></a> <span class="n">imports</span> <span class="o">=</span> <span class="p">(</span>
</span><span id="L-339"><a href="#L-339"><span class="linenos">339</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;&lt;script type=&quot;text/javascript&quot; src=&quot;https://unpkg.com/vis-data@latest/peer/umd/vis-data.min.js&quot;&gt;&lt;/script&gt;</span>
</span><span id="L-340"><a href="#L-340"><span class="linenos">340</span></a><span class="sd"> &lt;script type=&quot;text/javascript&quot; src=&quot;https://unpkg.com/vis-network@latest/peer/umd/vis-network.min.js&quot;&gt;&lt;/script&gt;</span>
</span><span id="L-341"><a href="#L-341"><span class="linenos">341</span></a><span class="sd"> &lt;link rel=&quot;stylesheet&quot; type=&quot;text/css&quot; href=&quot;https://unpkg.com/vis-network/styles/vis-network.min.css&quot; /&gt;&quot;&quot;&quot;</span>
</span><span id="L-342"><a href="#L-342"><span class="linenos">342</span></a> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span>
</span><span id="L-343"><a href="#L-343"><span class="linenos">343</span></a> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
</span><span id="L-344"><a href="#L-344"><span class="linenos">344</span></a> <span class="p">)</span>
</span><span id="L-345"><a href="#L-345"><span class="linenos">345</span></a>
</span><span id="L-346"><a href="#L-346"><span class="linenos">346</span></a> <span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;&quot;&quot;&lt;div&gt;</span>
</span><span id="L-347"><a href="#L-347"><span class="linenos">347</span></a><span class="s2"> &lt;div id=&quot;sqlglot-lineage&quot;&gt;&lt;/div&gt;</span>
</span><span id="L-348"><a href="#L-348"><span class="linenos">348</span></a><span class="s2"> </span><span class="si">{</span><span class="n">imports</span><span class="si">}</span>
</span><span id="L-349"><a href="#L-349"><span class="linenos">349</span></a><span class="s2"> &lt;script type=&quot;text/javascript&quot;&gt;</span>
</span><span id="L-350"><a href="#L-350"><span class="linenos">350</span></a><span class="s2"> var nodes = new vis.DataSet(</span><span class="si">{</span><span class="n">nodes</span><span class="si">}</span><span class="s2">)</span>
</span><span id="L-351"><a href="#L-351"><span class="linenos">351</span></a><span class="s2"> nodes.forEach(row =&gt; row[&quot;title&quot;] = new DOMParser().parseFromString(row[&quot;title&quot;], &quot;text/html&quot;).body.childNodes[0])</span>
</span><span id="L-352"><a href="#L-352"><span class="linenos">352</span></a>
</span><span id="L-353"><a href="#L-353"><span class="linenos">353</span></a><span class="s2"> new vis.Network(</span>
</span><span id="L-354"><a href="#L-354"><span class="linenos">354</span></a><span class="s2"> document.getElementById(&quot;sqlglot-lineage&quot;),</span>
</span><span id="L-355"><a href="#L-355"><span class="linenos">355</span></a><span class="s2"> </span><span class="se">{{</span>
</span><span id="L-356"><a href="#L-356"><span class="linenos">356</span></a><span class="s2"> nodes: nodes,</span>
</span><span id="L-357"><a href="#L-357"><span class="linenos">357</span></a><span class="s2"> edges: new vis.DataSet(</span><span class="si">{</span><span class="n">edges</span><span class="si">}</span><span class="s2">)</span>
</span><span id="L-358"><a href="#L-358"><span class="linenos">358</span></a><span class="s2"> </span><span class="se">}}</span><span class="s2">,</span>
</span><span id="L-359"><a href="#L-359"><span class="linenos">359</span></a><span class="s2"> </span><span class="si">{</span><span class="n">options</span><span class="si">}</span><span class="s2">,</span>
</span><span id="L-360"><a href="#L-360"><span class="linenos">360</span></a><span class="s2"> )</span>
</span><span id="L-361"><a href="#L-361"><span class="linenos">361</span></a><span class="s2"> &lt;/script&gt;</span>
</span><span id="L-362"><a href="#L-362"><span class="linenos">362</span></a><span class="s2">&lt;/div&gt;&quot;&quot;&quot;</span>
</span><span id="L-363"><a href="#L-363"><span class="linenos">363</span></a>
</span><span id="L-364"><a href="#L-364"><span class="linenos">364</span></a> <span class="k">def</span> <span class="nf">_repr_html_</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span><span id="L-365"><a href="#L-365"><span class="linenos">365</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span>
</span></pre></div>
@ -920,58 +924,62 @@
</span><span id="to_node-230"><a href="#to_node-230"><span class="linenos">230</span></a> <span class="k">for</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
</span><span id="to_node-231"><a href="#to_node-231"><span class="linenos">231</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">):</span>
</span><span id="to_node-232"><a href="#to_node-232"><span class="linenos">232</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">source</span><span class="o">.</span><span class="n">expression</span>
</span><span id="to_node-233"><a href="#to_node-233"><span class="linenos">233</span></a> <span class="n">node</span><span class="o">.</span><span class="n">downstream</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">select</span><span class="o">.</span><span class="n">sql</span><span class="p">(),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">source</span><span class="p">))</span>
</span><span id="to_node-234"><a href="#to_node-234"><span class="linenos">234</span></a>
</span><span id="to_node-235"><a href="#to_node-235"><span class="linenos">235</span></a> <span class="c1"># Find all columns that went into creating this one to list their lineage nodes.</span>
</span><span id="to_node-236"><a href="#to_node-236"><span class="linenos">236</span></a> <span class="n">source_columns</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">find_all_in_scope</span><span class="p">(</span><span class="n">select</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Column</span><span class="p">))</span>
</span><span id="to_node-237"><a href="#to_node-237"><span class="linenos">237</span></a>
</span><span id="to_node-238"><a href="#to_node-238"><span class="linenos">238</span></a> <span class="c1"># If the source is a UDTF find columns used in the UTDF to generate the table</span>
</span><span id="to_node-239"><a href="#to_node-239"><span class="linenos">239</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">):</span>
</span><span id="to_node-240"><a href="#to_node-240"><span class="linenos">240</span></a> <span class="n">source_columns</span> <span class="o">|=</span> <span class="nb">set</span><span class="p">(</span><span class="n">source</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Column</span><span class="p">))</span>
</span><span id="to_node-241"><a href="#to_node-241"><span class="linenos">241</span></a> <span class="n">derived_tables</span> <span class="o">=</span> <span class="p">[</span>
</span><span id="to_node-242"><a href="#to_node-242"><span class="linenos">242</span></a> <span class="n">source</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span>
</span><span id="to_node-243"><a href="#to_node-243"><span class="linenos">243</span></a> <span class="k">for</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
</span><span id="to_node-244"><a href="#to_node-244"><span class="linenos">244</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">)</span> <span class="ow">and</span> <span class="n">source</span><span class="o">.</span><span class="n">is_derived_table</span>
</span><span id="to_node-245"><a href="#to_node-245"><span class="linenos">245</span></a> <span class="p">]</span>
</span><span id="to_node-246"><a href="#to_node-246"><span class="linenos">246</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="to_node-247"><a href="#to_node-247"><span class="linenos">247</span></a> <span class="n">derived_tables</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">derived_tables</span>
</span><span id="to_node-248"><a href="#to_node-248"><span class="linenos">248</span></a>
</span><span id="to_node-249"><a href="#to_node-249"><span class="linenos">249</span></a> <span class="n">source_names</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="to_node-250"><a href="#to_node-250"><span class="linenos">250</span></a> <span class="n">dt</span><span class="o">.</span><span class="n">alias</span><span class="p">:</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span>
</span><span id="to_node-251"><a href="#to_node-251"><span class="linenos">251</span></a> <span class="k">for</span> <span class="n">dt</span> <span class="ow">in</span> <span class="n">derived_tables</span>
</span><span id="to_node-252"><a href="#to_node-252"><span class="linenos">252</span></a> <span class="k">if</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span> <span class="ow">and</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;source: &quot;</span><span class="p">)</span>
</span><span id="to_node-253"><a href="#to_node-253"><span class="linenos">253</span></a> <span class="p">}</span>
</span><span id="to_node-254"><a href="#to_node-254"><span class="linenos">254</span></a>
</span><span id="to_node-255"><a href="#to_node-255"><span class="linenos">255</span></a> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">source_columns</span><span class="p">:</span>
</span><span id="to_node-256"><a href="#to_node-256"><span class="linenos">256</span></a> <span class="n">table</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="n">table</span>
</span><span id="to_node-257"><a href="#to_node-257"><span class="linenos">257</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">)</span>
</span><span id="to_node-258"><a href="#to_node-258"><span class="linenos">258</span></a>
</span><span id="to_node-259"><a href="#to_node-259"><span class="linenos">259</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">):</span>
</span><span id="to_node-260"><a href="#to_node-260"><span class="linenos">260</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="to_node-261"><a href="#to_node-261"><span class="linenos">261</span></a> <span class="k">if</span> <span class="n">source</span><span class="o">.</span><span class="n">scope_type</span> <span class="o">==</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">DERIVED_TABLE</span> <span class="ow">and</span> <span class="n">table</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">source_names</span><span class="p">:</span>
</span><span id="to_node-262"><a href="#to_node-262"><span class="linenos">262</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="n">table</span>
</span><span id="to_node-263"><a href="#to_node-263"><span class="linenos">263</span></a> <span class="k">elif</span> <span class="n">source</span><span class="o">.</span><span class="n">scope_type</span> <span class="o">==</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">CTE</span><span class="p">:</span>
</span><span id="to_node-264"><a href="#to_node-264"><span class="linenos">264</span></a> <span class="n">selected_node</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">selected_sources</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">))</span>
</span><span id="to_node-265"><a href="#to_node-265"><span class="linenos">265</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="n">selected_node</span><span class="o">.</span><span class="n">name</span> <span class="k">if</span> <span class="n">selected_node</span> <span class="k">else</span> <span class="kc">None</span>
</span><span id="to_node-266"><a href="#to_node-266"><span class="linenos">266</span></a> <span class="c1"># The table itself came from a more specific scope. Recurse into that one using the unaliased column name.</span>
</span><span id="to_node-267"><a href="#to_node-267"><span class="linenos">267</span></a> <span class="n">to_node</span><span class="p">(</span>
</span><span id="to_node-268"><a href="#to_node-268"><span class="linenos">268</span></a> <span class="n">c</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
</span><span id="to_node-269"><a href="#to_node-269"><span class="linenos">269</span></a> <span class="n">scope</span><span class="o">=</span><span class="n">source</span><span class="p">,</span>
</span><span id="to_node-270"><a href="#to_node-270"><span class="linenos">270</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="to_node-271"><a href="#to_node-271"><span class="linenos">271</span></a> <span class="n">scope_name</span><span class="o">=</span><span class="n">table</span><span class="p">,</span>
</span><span id="to_node-272"><a href="#to_node-272"><span class="linenos">272</span></a> <span class="n">upstream</span><span class="o">=</span><span class="n">node</span><span class="p">,</span>
</span><span id="to_node-273"><a href="#to_node-273"><span class="linenos">273</span></a> <span class="n">source_name</span><span class="o">=</span><span class="n">source_names</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">)</span> <span class="ow">or</span> <span class="n">source_name</span><span class="p">,</span>
</span><span id="to_node-274"><a href="#to_node-274"><span class="linenos">274</span></a> <span class="n">reference_node_name</span><span class="o">=</span><span class="n">reference_node_name</span><span class="p">,</span>
</span><span id="to_node-275"><a href="#to_node-275"><span class="linenos">275</span></a> <span class="n">trim_selects</span><span class="o">=</span><span class="n">trim_selects</span><span class="p">,</span>
</span><span id="to_node-276"><a href="#to_node-276"><span class="linenos">276</span></a> <span class="p">)</span>
</span><span id="to_node-277"><a href="#to_node-277"><span class="linenos">277</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="to_node-278"><a href="#to_node-278"><span class="linenos">278</span></a> <span class="c1"># The source is not a scope - we&#39;ve reached the end of the line. At this point, if a source is not found</span>
</span><span id="to_node-279"><a href="#to_node-279"><span class="linenos">279</span></a> <span class="c1"># it means this column&#39;s lineage is unknown. This can happen if the definition of a source used in a query</span>
</span><span id="to_node-280"><a href="#to_node-280"><span class="linenos">280</span></a> <span class="c1"># is not passed into the `sources` map.</span>
</span><span id="to_node-281"><a href="#to_node-281"><span class="linenos">281</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">source</span> <span class="ow">or</span> <span class="n">exp</span><span class="o">.</span><span class="n">Placeholder</span><span class="p">()</span>
</span><span id="to_node-282"><a href="#to_node-282"><span class="linenos">282</span></a> <span class="n">node</span><span class="o">.</span><span class="n">downstream</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">c</span><span class="o">.</span><span class="n">sql</span><span class="p">(),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">source</span><span class="p">))</span>
</span><span id="to_node-283"><a href="#to_node-283"><span class="linenos">283</span></a>
</span><span id="to_node-284"><a href="#to_node-284"><span class="linenos">284</span></a> <span class="k">return</span> <span class="n">node</span>
</span><span id="to_node-233"><a href="#to_node-233"><span class="linenos">233</span></a> <span class="n">node</span><span class="o">.</span><span class="n">downstream</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
</span><span id="to_node-234"><a href="#to_node-234"><span class="linenos">234</span></a> <span class="n">Node</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">select</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="n">comments</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">source</span><span class="p">)</span>
</span><span id="to_node-235"><a href="#to_node-235"><span class="linenos">235</span></a> <span class="p">)</span>
</span><span id="to_node-236"><a href="#to_node-236"><span class="linenos">236</span></a>
</span><span id="to_node-237"><a href="#to_node-237"><span class="linenos">237</span></a> <span class="c1"># Find all columns that went into creating this one to list their lineage nodes.</span>
</span><span id="to_node-238"><a href="#to_node-238"><span class="linenos">238</span></a> <span class="n">source_columns</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">find_all_in_scope</span><span class="p">(</span><span class="n">select</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Column</span><span class="p">))</span>
</span><span id="to_node-239"><a href="#to_node-239"><span class="linenos">239</span></a>
</span><span id="to_node-240"><a href="#to_node-240"><span class="linenos">240</span></a> <span class="c1"># If the source is a UDTF find columns used in the UTDF to generate the table</span>
</span><span id="to_node-241"><a href="#to_node-241"><span class="linenos">241</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">):</span>
</span><span id="to_node-242"><a href="#to_node-242"><span class="linenos">242</span></a> <span class="n">source_columns</span> <span class="o">|=</span> <span class="nb">set</span><span class="p">(</span><span class="n">source</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Column</span><span class="p">))</span>
</span><span id="to_node-243"><a href="#to_node-243"><span class="linenos">243</span></a> <span class="n">derived_tables</span> <span class="o">=</span> <span class="p">[</span>
</span><span id="to_node-244"><a href="#to_node-244"><span class="linenos">244</span></a> <span class="n">source</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span>
</span><span id="to_node-245"><a href="#to_node-245"><span class="linenos">245</span></a> <span class="k">for</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
</span><span id="to_node-246"><a href="#to_node-246"><span class="linenos">246</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">)</span> <span class="ow">and</span> <span class="n">source</span><span class="o">.</span><span class="n">is_derived_table</span>
</span><span id="to_node-247"><a href="#to_node-247"><span class="linenos">247</span></a> <span class="p">]</span>
</span><span id="to_node-248"><a href="#to_node-248"><span class="linenos">248</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="to_node-249"><a href="#to_node-249"><span class="linenos">249</span></a> <span class="n">derived_tables</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">derived_tables</span>
</span><span id="to_node-250"><a href="#to_node-250"><span class="linenos">250</span></a>
</span><span id="to_node-251"><a href="#to_node-251"><span class="linenos">251</span></a> <span class="n">source_names</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="to_node-252"><a href="#to_node-252"><span class="linenos">252</span></a> <span class="n">dt</span><span class="o">.</span><span class="n">alias</span><span class="p">:</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span>
</span><span id="to_node-253"><a href="#to_node-253"><span class="linenos">253</span></a> <span class="k">for</span> <span class="n">dt</span> <span class="ow">in</span> <span class="n">derived_tables</span>
</span><span id="to_node-254"><a href="#to_node-254"><span class="linenos">254</span></a> <span class="k">if</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span> <span class="ow">and</span> <span class="n">dt</span><span class="o">.</span><span class="n">comments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;source: &quot;</span><span class="p">)</span>
</span><span id="to_node-255"><a href="#to_node-255"><span class="linenos">255</span></a> <span class="p">}</span>
</span><span id="to_node-256"><a href="#to_node-256"><span class="linenos">256</span></a>
</span><span id="to_node-257"><a href="#to_node-257"><span class="linenos">257</span></a> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">source_columns</span><span class="p">:</span>
</span><span id="to_node-258"><a href="#to_node-258"><span class="linenos">258</span></a> <span class="n">table</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="n">table</span>
</span><span id="to_node-259"><a href="#to_node-259"><span class="linenos">259</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">)</span>
</span><span id="to_node-260"><a href="#to_node-260"><span class="linenos">260</span></a>
</span><span id="to_node-261"><a href="#to_node-261"><span class="linenos">261</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">Scope</span><span class="p">):</span>
</span><span id="to_node-262"><a href="#to_node-262"><span class="linenos">262</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="to_node-263"><a href="#to_node-263"><span class="linenos">263</span></a> <span class="k">if</span> <span class="n">source</span><span class="o">.</span><span class="n">scope_type</span> <span class="o">==</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">DERIVED_TABLE</span> <span class="ow">and</span> <span class="n">table</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">source_names</span><span class="p">:</span>
</span><span id="to_node-264"><a href="#to_node-264"><span class="linenos">264</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="n">table</span>
</span><span id="to_node-265"><a href="#to_node-265"><span class="linenos">265</span></a> <span class="k">elif</span> <span class="n">source</span><span class="o">.</span><span class="n">scope_type</span> <span class="o">==</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">CTE</span><span class="p">:</span>
</span><span id="to_node-266"><a href="#to_node-266"><span class="linenos">266</span></a> <span class="n">selected_node</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">selected_sources</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">))</span>
</span><span id="to_node-267"><a href="#to_node-267"><span class="linenos">267</span></a> <span class="n">reference_node_name</span> <span class="o">=</span> <span class="n">selected_node</span><span class="o">.</span><span class="n">name</span> <span class="k">if</span> <span class="n">selected_node</span> <span class="k">else</span> <span class="kc">None</span>
</span><span id="to_node-268"><a href="#to_node-268"><span class="linenos">268</span></a> <span class="c1"># The table itself came from a more specific scope. Recurse into that one using the unaliased column name.</span>
</span><span id="to_node-269"><a href="#to_node-269"><span class="linenos">269</span></a> <span class="n">to_node</span><span class="p">(</span>
</span><span id="to_node-270"><a href="#to_node-270"><span class="linenos">270</span></a> <span class="n">c</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
</span><span id="to_node-271"><a href="#to_node-271"><span class="linenos">271</span></a> <span class="n">scope</span><span class="o">=</span><span class="n">source</span><span class="p">,</span>
</span><span id="to_node-272"><a href="#to_node-272"><span class="linenos">272</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="to_node-273"><a href="#to_node-273"><span class="linenos">273</span></a> <span class="n">scope_name</span><span class="o">=</span><span class="n">table</span><span class="p">,</span>
</span><span id="to_node-274"><a href="#to_node-274"><span class="linenos">274</span></a> <span class="n">upstream</span><span class="o">=</span><span class="n">node</span><span class="p">,</span>
</span><span id="to_node-275"><a href="#to_node-275"><span class="linenos">275</span></a> <span class="n">source_name</span><span class="o">=</span><span class="n">source_names</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">table</span><span class="p">)</span> <span class="ow">or</span> <span class="n">source_name</span><span class="p">,</span>
</span><span id="to_node-276"><a href="#to_node-276"><span class="linenos">276</span></a> <span class="n">reference_node_name</span><span class="o">=</span><span class="n">reference_node_name</span><span class="p">,</span>
</span><span id="to_node-277"><a href="#to_node-277"><span class="linenos">277</span></a> <span class="n">trim_selects</span><span class="o">=</span><span class="n">trim_selects</span><span class="p">,</span>
</span><span id="to_node-278"><a href="#to_node-278"><span class="linenos">278</span></a> <span class="p">)</span>
</span><span id="to_node-279"><a href="#to_node-279"><span class="linenos">279</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="to_node-280"><a href="#to_node-280"><span class="linenos">280</span></a> <span class="c1"># The source is not a scope - we&#39;ve reached the end of the line. At this point, if a source is not found</span>
</span><span id="to_node-281"><a href="#to_node-281"><span class="linenos">281</span></a> <span class="c1"># it means this column&#39;s lineage is unknown. This can happen if the definition of a source used in a query</span>
</span><span id="to_node-282"><a href="#to_node-282"><span class="linenos">282</span></a> <span class="c1"># is not passed into the `sources` map.</span>
</span><span id="to_node-283"><a href="#to_node-283"><span class="linenos">283</span></a> <span class="n">source</span> <span class="o">=</span> <span class="n">source</span> <span class="ow">or</span> <span class="n">exp</span><span class="o">.</span><span class="n">Placeholder</span><span class="p">()</span>
</span><span id="to_node-284"><a href="#to_node-284"><span class="linenos">284</span></a> <span class="n">node</span><span class="o">.</span><span class="n">downstream</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
</span><span id="to_node-285"><a href="#to_node-285"><span class="linenos">285</span></a> <span class="n">Node</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">c</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="n">comments</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">source</span><span class="p">)</span>
</span><span id="to_node-286"><a href="#to_node-286"><span class="linenos">286</span></a> <span class="p">)</span>
</span><span id="to_node-287"><a href="#to_node-287"><span class="linenos">287</span></a>
</span><span id="to_node-288"><a href="#to_node-288"><span class="linenos">288</span></a> <span class="k">return</span> <span class="n">node</span>
</span></pre></div>
@ -989,82 +997,82 @@
</div>
<a class="headerlink" href="#GraphHTML"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="GraphHTML-287"><a href="#GraphHTML-287"><span class="linenos">287</span></a><span class="k">class</span> <span class="nc">GraphHTML</span><span class="p">:</span>
</span><span id="GraphHTML-288"><a href="#GraphHTML-288"><span class="linenos">288</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;Node to HTML generator using vis.js.</span>
</span><span id="GraphHTML-289"><a href="#GraphHTML-289"><span class="linenos">289</span></a>
</span><span id="GraphHTML-290"><a href="#GraphHTML-290"><span class="linenos">290</span></a><span class="sd"> https://visjs.github.io/vis-network/docs/network/</span>
</span><span id="GraphHTML-291"><a href="#GraphHTML-291"><span class="linenos">291</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="GraphHTML-292"><a href="#GraphHTML-292"><span class="linenos">292</span></a>
</span><span id="GraphHTML-293"><a href="#GraphHTML-293"><span class="linenos">293</span></a> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span>
</span><span id="GraphHTML-294"><a href="#GraphHTML-294"><span class="linenos">294</span></a> <span class="bp">self</span><span class="p">,</span> <span class="n">nodes</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">edges</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">,</span> <span class="n">imports</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span> <span class="n">options</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="GraphHTML-295"><a href="#GraphHTML-295"><span class="linenos">295</span></a> <span class="p">):</span>
</span><span id="GraphHTML-296"><a href="#GraphHTML-296"><span class="linenos">296</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span> <span class="o">=</span> <span class="n">imports</span>
</span><span id="GraphHTML-297"><a href="#GraphHTML-297"><span class="linenos">297</span></a>
</span><span id="GraphHTML-298"><a href="#GraphHTML-298"><span class="linenos">298</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="GraphHTML-299"><a href="#GraphHTML-299"><span class="linenos">299</span></a> <span class="s2">&quot;height&quot;</span><span class="p">:</span> <span class="s2">&quot;500px&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-300"><a href="#GraphHTML-300"><span class="linenos">300</span></a> <span class="s2">&quot;width&quot;</span><span class="p">:</span> <span class="s2">&quot;100%&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-301"><a href="#GraphHTML-301"><span class="linenos">301</span></a> <span class="s2">&quot;layout&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-302"><a href="#GraphHTML-302"><span class="linenos">302</span></a> <span class="s2">&quot;hierarchical&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-303"><a href="#GraphHTML-303"><span class="linenos">303</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="GraphHTML-304"><a href="#GraphHTML-304"><span class="linenos">304</span></a> <span class="s2">&quot;nodeSpacing&quot;</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
</span><span id="GraphHTML-305"><a href="#GraphHTML-305"><span class="linenos">305</span></a> <span class="s2">&quot;sortMethod&quot;</span><span class="p">:</span> <span class="s2">&quot;directed&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-306"><a href="#GraphHTML-306"><span class="linenos">306</span></a> <span class="p">},</span>
</span><span id="GraphHTML-307"><a href="#GraphHTML-307"><span class="linenos">307</span></a> <span class="p">},</span>
</span><span id="GraphHTML-308"><a href="#GraphHTML-308"><span class="linenos">308</span></a> <span class="s2">&quot;interaction&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-309"><a href="#GraphHTML-309"><span class="linenos">309</span></a> <span class="s2">&quot;dragNodes&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML-310"><a href="#GraphHTML-310"><span class="linenos">310</span></a> <span class="s2">&quot;selectable&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="GraphHTML-291"><a href="#GraphHTML-291"><span class="linenos">291</span></a><span class="k">class</span> <span class="nc">GraphHTML</span><span class="p">:</span>
</span><span id="GraphHTML-292"><a href="#GraphHTML-292"><span class="linenos">292</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;Node to HTML generator using vis.js.</span>
</span><span id="GraphHTML-293"><a href="#GraphHTML-293"><span class="linenos">293</span></a>
</span><span id="GraphHTML-294"><a href="#GraphHTML-294"><span class="linenos">294</span></a><span class="sd"> https://visjs.github.io/vis-network/docs/network/</span>
</span><span id="GraphHTML-295"><a href="#GraphHTML-295"><span class="linenos">295</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="GraphHTML-296"><a href="#GraphHTML-296"><span class="linenos">296</span></a>
</span><span id="GraphHTML-297"><a href="#GraphHTML-297"><span class="linenos">297</span></a> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span>
</span><span id="GraphHTML-298"><a href="#GraphHTML-298"><span class="linenos">298</span></a> <span class="bp">self</span><span class="p">,</span> <span class="n">nodes</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">edges</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">,</span> <span class="n">imports</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span> <span class="n">options</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="GraphHTML-299"><a href="#GraphHTML-299"><span class="linenos">299</span></a> <span class="p">):</span>
</span><span id="GraphHTML-300"><a href="#GraphHTML-300"><span class="linenos">300</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span> <span class="o">=</span> <span class="n">imports</span>
</span><span id="GraphHTML-301"><a href="#GraphHTML-301"><span class="linenos">301</span></a>
</span><span id="GraphHTML-302"><a href="#GraphHTML-302"><span class="linenos">302</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="GraphHTML-303"><a href="#GraphHTML-303"><span class="linenos">303</span></a> <span class="s2">&quot;height&quot;</span><span class="p">:</span> <span class="s2">&quot;500px&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-304"><a href="#GraphHTML-304"><span class="linenos">304</span></a> <span class="s2">&quot;width&quot;</span><span class="p">:</span> <span class="s2">&quot;100%&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-305"><a href="#GraphHTML-305"><span class="linenos">305</span></a> <span class="s2">&quot;layout&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-306"><a href="#GraphHTML-306"><span class="linenos">306</span></a> <span class="s2">&quot;hierarchical&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-307"><a href="#GraphHTML-307"><span class="linenos">307</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="GraphHTML-308"><a href="#GraphHTML-308"><span class="linenos">308</span></a> <span class="s2">&quot;nodeSpacing&quot;</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
</span><span id="GraphHTML-309"><a href="#GraphHTML-309"><span class="linenos">309</span></a> <span class="s2">&quot;sortMethod&quot;</span><span class="p">:</span> <span class="s2">&quot;directed&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-310"><a href="#GraphHTML-310"><span class="linenos">310</span></a> <span class="p">},</span>
</span><span id="GraphHTML-311"><a href="#GraphHTML-311"><span class="linenos">311</span></a> <span class="p">},</span>
</span><span id="GraphHTML-312"><a href="#GraphHTML-312"><span class="linenos">312</span></a> <span class="s2">&quot;physics&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-313"><a href="#GraphHTML-313"><span class="linenos">313</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML-314"><a href="#GraphHTML-314"><span class="linenos">314</span></a> <span class="p">},</span>
</span><span id="GraphHTML-315"><a href="#GraphHTML-315"><span class="linenos">315</span></a> <span class="s2">&quot;edges&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-316"><a href="#GraphHTML-316"><span class="linenos">316</span></a> <span class="s2">&quot;arrows&quot;</span><span class="p">:</span> <span class="s2">&quot;to&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-317"><a href="#GraphHTML-317"><span class="linenos">317</span></a> <span class="p">},</span>
</span><span id="GraphHTML-318"><a href="#GraphHTML-318"><span class="linenos">318</span></a> <span class="s2">&quot;nodes&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-319"><a href="#GraphHTML-319"><span class="linenos">319</span></a> <span class="s2">&quot;font&quot;</span><span class="p">:</span> <span class="s2">&quot;20px monaco&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-320"><a href="#GraphHTML-320"><span class="linenos">320</span></a> <span class="s2">&quot;shape&quot;</span><span class="p">:</span> <span class="s2">&quot;box&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-321"><a href="#GraphHTML-321"><span class="linenos">321</span></a> <span class="s2">&quot;widthConstraint&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-322"><a href="#GraphHTML-322"><span class="linenos">322</span></a> <span class="s2">&quot;maximum&quot;</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
</span><span id="GraphHTML-323"><a href="#GraphHTML-323"><span class="linenos">323</span></a> <span class="p">},</span>
</span><span id="GraphHTML-324"><a href="#GraphHTML-324"><span class="linenos">324</span></a> <span class="p">},</span>
</span><span id="GraphHTML-325"><a href="#GraphHTML-325"><span class="linenos">325</span></a> <span class="o">**</span><span class="p">(</span><span class="n">options</span> <span class="ow">or</span> <span class="p">{}),</span>
</span><span id="GraphHTML-326"><a href="#GraphHTML-326"><span class="linenos">326</span></a> <span class="p">}</span>
</span><span id="GraphHTML-327"><a href="#GraphHTML-327"><span class="linenos">327</span></a>
</span><span id="GraphHTML-328"><a href="#GraphHTML-328"><span class="linenos">328</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">nodes</span> <span class="o">=</span> <span class="n">nodes</span>
</span><span id="GraphHTML-329"><a href="#GraphHTML-329"><span class="linenos">329</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">edges</span> <span class="o">=</span> <span class="n">edges</span>
</span><span id="GraphHTML-330"><a href="#GraphHTML-330"><span class="linenos">330</span></a>
</span><span id="GraphHTML-331"><a href="#GraphHTML-331"><span class="linenos">331</span></a> <span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span><span id="GraphHTML-332"><a href="#GraphHTML-332"><span class="linenos">332</span></a> <span class="n">nodes</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">nodes</span><span class="o">.</span><span class="n">values</span><span class="p">()))</span>
</span><span id="GraphHTML-333"><a href="#GraphHTML-333"><span class="linenos">333</span></a> <span class="n">edges</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">edges</span><span class="p">)</span>
</span><span id="GraphHTML-334"><a href="#GraphHTML-334"><span class="linenos">334</span></a> <span class="n">options</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">options</span><span class="p">)</span>
</span><span id="GraphHTML-335"><a href="#GraphHTML-335"><span class="linenos">335</span></a> <span class="n">imports</span> <span class="o">=</span> <span class="p">(</span>
</span><span id="GraphHTML-336"><a href="#GraphHTML-336"><span class="linenos">336</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;&lt;script type=&quot;text/javascript&quot; src=&quot;https://unpkg.com/vis-data@latest/peer/umd/vis-data.min.js&quot;&gt;&lt;/script&gt;</span>
</span><span id="GraphHTML-337"><a href="#GraphHTML-337"><span class="linenos">337</span></a><span class="sd"> &lt;script type=&quot;text/javascript&quot; src=&quot;https://unpkg.com/vis-network@latest/peer/umd/vis-network.min.js&quot;&gt;&lt;/script&gt;</span>
</span><span id="GraphHTML-338"><a href="#GraphHTML-338"><span class="linenos">338</span></a><span class="sd"> &lt;link rel=&quot;stylesheet&quot; type=&quot;text/css&quot; href=&quot;https://unpkg.com/vis-network/styles/vis-network.min.css&quot; /&gt;&quot;&quot;&quot;</span>
</span><span id="GraphHTML-339"><a href="#GraphHTML-339"><span class="linenos">339</span></a> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span>
</span><span id="GraphHTML-340"><a href="#GraphHTML-340"><span class="linenos">340</span></a> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
</span><span id="GraphHTML-341"><a href="#GraphHTML-341"><span class="linenos">341</span></a> <span class="p">)</span>
</span><span id="GraphHTML-342"><a href="#GraphHTML-342"><span class="linenos">342</span></a>
</span><span id="GraphHTML-343"><a href="#GraphHTML-343"><span class="linenos">343</span></a> <span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;&quot;&quot;&lt;div&gt;</span>
</span><span id="GraphHTML-344"><a href="#GraphHTML-344"><span class="linenos">344</span></a><span class="s2"> &lt;div id=&quot;sqlglot-lineage&quot;&gt;&lt;/div&gt;</span>
</span><span id="GraphHTML-345"><a href="#GraphHTML-345"><span class="linenos">345</span></a><span class="s2"> </span><span class="si">{</span><span class="n">imports</span><span class="si">}</span>
</span><span id="GraphHTML-346"><a href="#GraphHTML-346"><span class="linenos">346</span></a><span class="s2"> &lt;script type=&quot;text/javascript&quot;&gt;</span>
</span><span id="GraphHTML-347"><a href="#GraphHTML-347"><span class="linenos">347</span></a><span class="s2"> var nodes = new vis.DataSet(</span><span class="si">{</span><span class="n">nodes</span><span class="si">}</span><span class="s2">)</span>
</span><span id="GraphHTML-348"><a href="#GraphHTML-348"><span class="linenos">348</span></a><span class="s2"> nodes.forEach(row =&gt; row[&quot;title&quot;] = new DOMParser().parseFromString(row[&quot;title&quot;], &quot;text/html&quot;).body.childNodes[0])</span>
</span><span id="GraphHTML-349"><a href="#GraphHTML-349"><span class="linenos">349</span></a>
</span><span id="GraphHTML-350"><a href="#GraphHTML-350"><span class="linenos">350</span></a><span class="s2"> new vis.Network(</span>
</span><span id="GraphHTML-351"><a href="#GraphHTML-351"><span class="linenos">351</span></a><span class="s2"> document.getElementById(&quot;sqlglot-lineage&quot;),</span>
</span><span id="GraphHTML-352"><a href="#GraphHTML-352"><span class="linenos">352</span></a><span class="s2"> </span><span class="se">{{</span>
</span><span id="GraphHTML-353"><a href="#GraphHTML-353"><span class="linenos">353</span></a><span class="s2"> nodes: nodes,</span>
</span><span id="GraphHTML-354"><a href="#GraphHTML-354"><span class="linenos">354</span></a><span class="s2"> edges: new vis.DataSet(</span><span class="si">{</span><span class="n">edges</span><span class="si">}</span><span class="s2">)</span>
</span><span id="GraphHTML-355"><a href="#GraphHTML-355"><span class="linenos">355</span></a><span class="s2"> </span><span class="se">}}</span><span class="s2">,</span>
</span><span id="GraphHTML-356"><a href="#GraphHTML-356"><span class="linenos">356</span></a><span class="s2"> </span><span class="si">{</span><span class="n">options</span><span class="si">}</span><span class="s2">,</span>
</span><span id="GraphHTML-357"><a href="#GraphHTML-357"><span class="linenos">357</span></a><span class="s2"> )</span>
</span><span id="GraphHTML-358"><a href="#GraphHTML-358"><span class="linenos">358</span></a><span class="s2"> &lt;/script&gt;</span>
</span><span id="GraphHTML-359"><a href="#GraphHTML-359"><span class="linenos">359</span></a><span class="s2">&lt;/div&gt;&quot;&quot;&quot;</span>
</span><span id="GraphHTML-360"><a href="#GraphHTML-360"><span class="linenos">360</span></a>
</span><span id="GraphHTML-361"><a href="#GraphHTML-361"><span class="linenos">361</span></a> <span class="k">def</span> <span class="nf">_repr_html_</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span><span id="GraphHTML-362"><a href="#GraphHTML-362"><span class="linenos">362</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span>
</span><span id="GraphHTML-312"><a href="#GraphHTML-312"><span class="linenos">312</span></a> <span class="s2">&quot;interaction&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-313"><a href="#GraphHTML-313"><span class="linenos">313</span></a> <span class="s2">&quot;dragNodes&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML-314"><a href="#GraphHTML-314"><span class="linenos">314</span></a> <span class="s2">&quot;selectable&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML-315"><a href="#GraphHTML-315"><span class="linenos">315</span></a> <span class="p">},</span>
</span><span id="GraphHTML-316"><a href="#GraphHTML-316"><span class="linenos">316</span></a> <span class="s2">&quot;physics&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-317"><a href="#GraphHTML-317"><span class="linenos">317</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML-318"><a href="#GraphHTML-318"><span class="linenos">318</span></a> <span class="p">},</span>
</span><span id="GraphHTML-319"><a href="#GraphHTML-319"><span class="linenos">319</span></a> <span class="s2">&quot;edges&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-320"><a href="#GraphHTML-320"><span class="linenos">320</span></a> <span class="s2">&quot;arrows&quot;</span><span class="p">:</span> <span class="s2">&quot;to&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-321"><a href="#GraphHTML-321"><span class="linenos">321</span></a> <span class="p">},</span>
</span><span id="GraphHTML-322"><a href="#GraphHTML-322"><span class="linenos">322</span></a> <span class="s2">&quot;nodes&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-323"><a href="#GraphHTML-323"><span class="linenos">323</span></a> <span class="s2">&quot;font&quot;</span><span class="p">:</span> <span class="s2">&quot;20px monaco&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-324"><a href="#GraphHTML-324"><span class="linenos">324</span></a> <span class="s2">&quot;shape&quot;</span><span class="p">:</span> <span class="s2">&quot;box&quot;</span><span class="p">,</span>
</span><span id="GraphHTML-325"><a href="#GraphHTML-325"><span class="linenos">325</span></a> <span class="s2">&quot;widthConstraint&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML-326"><a href="#GraphHTML-326"><span class="linenos">326</span></a> <span class="s2">&quot;maximum&quot;</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
</span><span id="GraphHTML-327"><a href="#GraphHTML-327"><span class="linenos">327</span></a> <span class="p">},</span>
</span><span id="GraphHTML-328"><a href="#GraphHTML-328"><span class="linenos">328</span></a> <span class="p">},</span>
</span><span id="GraphHTML-329"><a href="#GraphHTML-329"><span class="linenos">329</span></a> <span class="o">**</span><span class="p">(</span><span class="n">options</span> <span class="ow">or</span> <span class="p">{}),</span>
</span><span id="GraphHTML-330"><a href="#GraphHTML-330"><span class="linenos">330</span></a> <span class="p">}</span>
</span><span id="GraphHTML-331"><a href="#GraphHTML-331"><span class="linenos">331</span></a>
</span><span id="GraphHTML-332"><a href="#GraphHTML-332"><span class="linenos">332</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">nodes</span> <span class="o">=</span> <span class="n">nodes</span>
</span><span id="GraphHTML-333"><a href="#GraphHTML-333"><span class="linenos">333</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">edges</span> <span class="o">=</span> <span class="n">edges</span>
</span><span id="GraphHTML-334"><a href="#GraphHTML-334"><span class="linenos">334</span></a>
</span><span id="GraphHTML-335"><a href="#GraphHTML-335"><span class="linenos">335</span></a> <span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span><span id="GraphHTML-336"><a href="#GraphHTML-336"><span class="linenos">336</span></a> <span class="n">nodes</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">nodes</span><span class="o">.</span><span class="n">values</span><span class="p">()))</span>
</span><span id="GraphHTML-337"><a href="#GraphHTML-337"><span class="linenos">337</span></a> <span class="n">edges</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">edges</span><span class="p">)</span>
</span><span id="GraphHTML-338"><a href="#GraphHTML-338"><span class="linenos">338</span></a> <span class="n">options</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">options</span><span class="p">)</span>
</span><span id="GraphHTML-339"><a href="#GraphHTML-339"><span class="linenos">339</span></a> <span class="n">imports</span> <span class="o">=</span> <span class="p">(</span>
</span><span id="GraphHTML-340"><a href="#GraphHTML-340"><span class="linenos">340</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;&lt;script type=&quot;text/javascript&quot; src=&quot;https://unpkg.com/vis-data@latest/peer/umd/vis-data.min.js&quot;&gt;&lt;/script&gt;</span>
</span><span id="GraphHTML-341"><a href="#GraphHTML-341"><span class="linenos">341</span></a><span class="sd"> &lt;script type=&quot;text/javascript&quot; src=&quot;https://unpkg.com/vis-network@latest/peer/umd/vis-network.min.js&quot;&gt;&lt;/script&gt;</span>
</span><span id="GraphHTML-342"><a href="#GraphHTML-342"><span class="linenos">342</span></a><span class="sd"> &lt;link rel=&quot;stylesheet&quot; type=&quot;text/css&quot; href=&quot;https://unpkg.com/vis-network/styles/vis-network.min.css&quot; /&gt;&quot;&quot;&quot;</span>
</span><span id="GraphHTML-343"><a href="#GraphHTML-343"><span class="linenos">343</span></a> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span>
</span><span id="GraphHTML-344"><a href="#GraphHTML-344"><span class="linenos">344</span></a> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
</span><span id="GraphHTML-345"><a href="#GraphHTML-345"><span class="linenos">345</span></a> <span class="p">)</span>
</span><span id="GraphHTML-346"><a href="#GraphHTML-346"><span class="linenos">346</span></a>
</span><span id="GraphHTML-347"><a href="#GraphHTML-347"><span class="linenos">347</span></a> <span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;&quot;&quot;&lt;div&gt;</span>
</span><span id="GraphHTML-348"><a href="#GraphHTML-348"><span class="linenos">348</span></a><span class="s2"> &lt;div id=&quot;sqlglot-lineage&quot;&gt;&lt;/div&gt;</span>
</span><span id="GraphHTML-349"><a href="#GraphHTML-349"><span class="linenos">349</span></a><span class="s2"> </span><span class="si">{</span><span class="n">imports</span><span class="si">}</span>
</span><span id="GraphHTML-350"><a href="#GraphHTML-350"><span class="linenos">350</span></a><span class="s2"> &lt;script type=&quot;text/javascript&quot;&gt;</span>
</span><span id="GraphHTML-351"><a href="#GraphHTML-351"><span class="linenos">351</span></a><span class="s2"> var nodes = new vis.DataSet(</span><span class="si">{</span><span class="n">nodes</span><span class="si">}</span><span class="s2">)</span>
</span><span id="GraphHTML-352"><a href="#GraphHTML-352"><span class="linenos">352</span></a><span class="s2"> nodes.forEach(row =&gt; row[&quot;title&quot;] = new DOMParser().parseFromString(row[&quot;title&quot;], &quot;text/html&quot;).body.childNodes[0])</span>
</span><span id="GraphHTML-353"><a href="#GraphHTML-353"><span class="linenos">353</span></a>
</span><span id="GraphHTML-354"><a href="#GraphHTML-354"><span class="linenos">354</span></a><span class="s2"> new vis.Network(</span>
</span><span id="GraphHTML-355"><a href="#GraphHTML-355"><span class="linenos">355</span></a><span class="s2"> document.getElementById(&quot;sqlglot-lineage&quot;),</span>
</span><span id="GraphHTML-356"><a href="#GraphHTML-356"><span class="linenos">356</span></a><span class="s2"> </span><span class="se">{{</span>
</span><span id="GraphHTML-357"><a href="#GraphHTML-357"><span class="linenos">357</span></a><span class="s2"> nodes: nodes,</span>
</span><span id="GraphHTML-358"><a href="#GraphHTML-358"><span class="linenos">358</span></a><span class="s2"> edges: new vis.DataSet(</span><span class="si">{</span><span class="n">edges</span><span class="si">}</span><span class="s2">)</span>
</span><span id="GraphHTML-359"><a href="#GraphHTML-359"><span class="linenos">359</span></a><span class="s2"> </span><span class="se">}}</span><span class="s2">,</span>
</span><span id="GraphHTML-360"><a href="#GraphHTML-360"><span class="linenos">360</span></a><span class="s2"> </span><span class="si">{</span><span class="n">options</span><span class="si">}</span><span class="s2">,</span>
</span><span id="GraphHTML-361"><a href="#GraphHTML-361"><span class="linenos">361</span></a><span class="s2"> )</span>
</span><span id="GraphHTML-362"><a href="#GraphHTML-362"><span class="linenos">362</span></a><span class="s2"> &lt;/script&gt;</span>
</span><span id="GraphHTML-363"><a href="#GraphHTML-363"><span class="linenos">363</span></a><span class="s2">&lt;/div&gt;&quot;&quot;&quot;</span>
</span><span id="GraphHTML-364"><a href="#GraphHTML-364"><span class="linenos">364</span></a>
</span><span id="GraphHTML-365"><a href="#GraphHTML-365"><span class="linenos">365</span></a> <span class="k">def</span> <span class="nf">_repr_html_</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span><span id="GraphHTML-366"><a href="#GraphHTML-366"><span class="linenos">366</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span>
</span></pre></div>
@ -1084,43 +1092,43 @@
</div>
<a class="headerlink" href="#GraphHTML.__init__"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="GraphHTML.__init__-293"><a href="#GraphHTML.__init__-293"><span class="linenos">293</span></a> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span>
</span><span id="GraphHTML.__init__-294"><a href="#GraphHTML.__init__-294"><span class="linenos">294</span></a> <span class="bp">self</span><span class="p">,</span> <span class="n">nodes</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">edges</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">,</span> <span class="n">imports</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span> <span class="n">options</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="GraphHTML.__init__-295"><a href="#GraphHTML.__init__-295"><span class="linenos">295</span></a> <span class="p">):</span>
</span><span id="GraphHTML.__init__-296"><a href="#GraphHTML.__init__-296"><span class="linenos">296</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span> <span class="o">=</span> <span class="n">imports</span>
</span><span id="GraphHTML.__init__-297"><a href="#GraphHTML.__init__-297"><span class="linenos">297</span></a>
</span><span id="GraphHTML.__init__-298"><a href="#GraphHTML.__init__-298"><span class="linenos">298</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-299"><a href="#GraphHTML.__init__-299"><span class="linenos">299</span></a> <span class="s2">&quot;height&quot;</span><span class="p">:</span> <span class="s2">&quot;500px&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-300"><a href="#GraphHTML.__init__-300"><span class="linenos">300</span></a> <span class="s2">&quot;width&quot;</span><span class="p">:</span> <span class="s2">&quot;100%&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-301"><a href="#GraphHTML.__init__-301"><span class="linenos">301</span></a> <span class="s2">&quot;layout&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-302"><a href="#GraphHTML.__init__-302"><span class="linenos">302</span></a> <span class="s2">&quot;hierarchical&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-303"><a href="#GraphHTML.__init__-303"><span class="linenos">303</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-304"><a href="#GraphHTML.__init__-304"><span class="linenos">304</span></a> <span class="s2">&quot;nodeSpacing&quot;</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-305"><a href="#GraphHTML.__init__-305"><span class="linenos">305</span></a> <span class="s2">&quot;sortMethod&quot;</span><span class="p">:</span> <span class="s2">&quot;directed&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-306"><a href="#GraphHTML.__init__-306"><span class="linenos">306</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-307"><a href="#GraphHTML.__init__-307"><span class="linenos">307</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-308"><a href="#GraphHTML.__init__-308"><span class="linenos">308</span></a> <span class="s2">&quot;interaction&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-309"><a href="#GraphHTML.__init__-309"><span class="linenos">309</span></a> <span class="s2">&quot;dragNodes&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-310"><a href="#GraphHTML.__init__-310"><span class="linenos">310</span></a> <span class="s2">&quot;selectable&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="GraphHTML.__init__-297"><a href="#GraphHTML.__init__-297"><span class="linenos">297</span></a> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span>
</span><span id="GraphHTML.__init__-298"><a href="#GraphHTML.__init__-298"><span class="linenos">298</span></a> <span class="bp">self</span><span class="p">,</span> <span class="n">nodes</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">edges</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">,</span> <span class="n">imports</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span> <span class="n">options</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="GraphHTML.__init__-299"><a href="#GraphHTML.__init__-299"><span class="linenos">299</span></a> <span class="p">):</span>
</span><span id="GraphHTML.__init__-300"><a href="#GraphHTML.__init__-300"><span class="linenos">300</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">imports</span> <span class="o">=</span> <span class="n">imports</span>
</span><span id="GraphHTML.__init__-301"><a href="#GraphHTML.__init__-301"><span class="linenos">301</span></a>
</span><span id="GraphHTML.__init__-302"><a href="#GraphHTML.__init__-302"><span class="linenos">302</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="o">=</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-303"><a href="#GraphHTML.__init__-303"><span class="linenos">303</span></a> <span class="s2">&quot;height&quot;</span><span class="p">:</span> <span class="s2">&quot;500px&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-304"><a href="#GraphHTML.__init__-304"><span class="linenos">304</span></a> <span class="s2">&quot;width&quot;</span><span class="p">:</span> <span class="s2">&quot;100%&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-305"><a href="#GraphHTML.__init__-305"><span class="linenos">305</span></a> <span class="s2">&quot;layout&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-306"><a href="#GraphHTML.__init__-306"><span class="linenos">306</span></a> <span class="s2">&quot;hierarchical&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-307"><a href="#GraphHTML.__init__-307"><span class="linenos">307</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-308"><a href="#GraphHTML.__init__-308"><span class="linenos">308</span></a> <span class="s2">&quot;nodeSpacing&quot;</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-309"><a href="#GraphHTML.__init__-309"><span class="linenos">309</span></a> <span class="s2">&quot;sortMethod&quot;</span><span class="p">:</span> <span class="s2">&quot;directed&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-310"><a href="#GraphHTML.__init__-310"><span class="linenos">310</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-311"><a href="#GraphHTML.__init__-311"><span class="linenos">311</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-312"><a href="#GraphHTML.__init__-312"><span class="linenos">312</span></a> <span class="s2">&quot;physics&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-313"><a href="#GraphHTML.__init__-313"><span class="linenos">313</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-314"><a href="#GraphHTML.__init__-314"><span class="linenos">314</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-315"><a href="#GraphHTML.__init__-315"><span class="linenos">315</span></a> <span class="s2">&quot;edges&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-316"><a href="#GraphHTML.__init__-316"><span class="linenos">316</span></a> <span class="s2">&quot;arrows&quot;</span><span class="p">:</span> <span class="s2">&quot;to&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-317"><a href="#GraphHTML.__init__-317"><span class="linenos">317</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-318"><a href="#GraphHTML.__init__-318"><span class="linenos">318</span></a> <span class="s2">&quot;nodes&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-319"><a href="#GraphHTML.__init__-319"><span class="linenos">319</span></a> <span class="s2">&quot;font&quot;</span><span class="p">:</span> <span class="s2">&quot;20px monaco&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-320"><a href="#GraphHTML.__init__-320"><span class="linenos">320</span></a> <span class="s2">&quot;shape&quot;</span><span class="p">:</span> <span class="s2">&quot;box&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-321"><a href="#GraphHTML.__init__-321"><span class="linenos">321</span></a> <span class="s2">&quot;widthConstraint&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-322"><a href="#GraphHTML.__init__-322"><span class="linenos">322</span></a> <span class="s2">&quot;maximum&quot;</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-323"><a href="#GraphHTML.__init__-323"><span class="linenos">323</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-324"><a href="#GraphHTML.__init__-324"><span class="linenos">324</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-325"><a href="#GraphHTML.__init__-325"><span class="linenos">325</span></a> <span class="o">**</span><span class="p">(</span><span class="n">options</span> <span class="ow">or</span> <span class="p">{}),</span>
</span><span id="GraphHTML.__init__-326"><a href="#GraphHTML.__init__-326"><span class="linenos">326</span></a> <span class="p">}</span>
</span><span id="GraphHTML.__init__-327"><a href="#GraphHTML.__init__-327"><span class="linenos">327</span></a>
</span><span id="GraphHTML.__init__-328"><a href="#GraphHTML.__init__-328"><span class="linenos">328</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">nodes</span> <span class="o">=</span> <span class="n">nodes</span>
</span><span id="GraphHTML.__init__-329"><a href="#GraphHTML.__init__-329"><span class="linenos">329</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">edges</span> <span class="o">=</span> <span class="n">edges</span>
</span><span id="GraphHTML.__init__-312"><a href="#GraphHTML.__init__-312"><span class="linenos">312</span></a> <span class="s2">&quot;interaction&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-313"><a href="#GraphHTML.__init__-313"><span class="linenos">313</span></a> <span class="s2">&quot;dragNodes&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-314"><a href="#GraphHTML.__init__-314"><span class="linenos">314</span></a> <span class="s2">&quot;selectable&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-315"><a href="#GraphHTML.__init__-315"><span class="linenos">315</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-316"><a href="#GraphHTML.__init__-316"><span class="linenos">316</span></a> <span class="s2">&quot;physics&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-317"><a href="#GraphHTML.__init__-317"><span class="linenos">317</span></a> <span class="s2">&quot;enabled&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-318"><a href="#GraphHTML.__init__-318"><span class="linenos">318</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-319"><a href="#GraphHTML.__init__-319"><span class="linenos">319</span></a> <span class="s2">&quot;edges&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-320"><a href="#GraphHTML.__init__-320"><span class="linenos">320</span></a> <span class="s2">&quot;arrows&quot;</span><span class="p">:</span> <span class="s2">&quot;to&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-321"><a href="#GraphHTML.__init__-321"><span class="linenos">321</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-322"><a href="#GraphHTML.__init__-322"><span class="linenos">322</span></a> <span class="s2">&quot;nodes&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-323"><a href="#GraphHTML.__init__-323"><span class="linenos">323</span></a> <span class="s2">&quot;font&quot;</span><span class="p">:</span> <span class="s2">&quot;20px monaco&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-324"><a href="#GraphHTML.__init__-324"><span class="linenos">324</span></a> <span class="s2">&quot;shape&quot;</span><span class="p">:</span> <span class="s2">&quot;box&quot;</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-325"><a href="#GraphHTML.__init__-325"><span class="linenos">325</span></a> <span class="s2">&quot;widthConstraint&quot;</span><span class="p">:</span> <span class="p">{</span>
</span><span id="GraphHTML.__init__-326"><a href="#GraphHTML.__init__-326"><span class="linenos">326</span></a> <span class="s2">&quot;maximum&quot;</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
</span><span id="GraphHTML.__init__-327"><a href="#GraphHTML.__init__-327"><span class="linenos">327</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-328"><a href="#GraphHTML.__init__-328"><span class="linenos">328</span></a> <span class="p">},</span>
</span><span id="GraphHTML.__init__-329"><a href="#GraphHTML.__init__-329"><span class="linenos">329</span></a> <span class="o">**</span><span class="p">(</span><span class="n">options</span> <span class="ow">or</span> <span class="p">{}),</span>
</span><span id="GraphHTML.__init__-330"><a href="#GraphHTML.__init__-330"><span class="linenos">330</span></a> <span class="p">}</span>
</span><span id="GraphHTML.__init__-331"><a href="#GraphHTML.__init__-331"><span class="linenos">331</span></a>
</span><span id="GraphHTML.__init__-332"><a href="#GraphHTML.__init__-332"><span class="linenos">332</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">nodes</span> <span class="o">=</span> <span class="n">nodes</span>
</span><span id="GraphHTML.__init__-333"><a href="#GraphHTML.__init__-333"><span class="linenos">333</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">edges</span> <span class="o">=</span> <span class="n">edges</span>
</span></pre></div>

8
docs/sqlglot/optimizer/annotate_types.html
View file

File diff suppressed because one or more lines are too long

2
docs/sqlglot/optimizer/merge_subqueries.html
View file

@ -586,7 +586,7 @@ queries if it would result in multiple table selects in a single query:</p>
<div class="attr variable">
<span class="name">UNMERGABLE_ARGS</span> =
<input id="UNMERGABLE_ARGS-view-value" class="view-value-toggle-state" type="checkbox" aria-hidden="true" tabindex="-1">
<label class="view-value-button pdoc-button" for="UNMERGABLE_ARGS-view-value"></label><span class="default_value">{&#39;with&#39;, &#39;locks&#39;, &#39;distinct&#39;, &#39;settings&#39;, &#39;offset&#39;, &#39;distribute&#39;, &#39;prewhere&#39;, &#39;laterals&#39;, &#39;qualify&#39;, &#39;windows&#39;, &#39;match&#39;, &#39;sample&#39;, &#39;sort&#39;, &#39;into&#39;, &#39;kind&#39;, &#39;pivots&#39;, &#39;limit&#39;, &#39;format&#39;, &#39;connect&#39;, &#39;cluster&#39;, &#39;options&#39;, &#39;group&#39;, &#39;having&#39;}</span>
<label class="view-value-button pdoc-button" for="UNMERGABLE_ARGS-view-value"></label><span class="default_value">{&#39;prewhere&#39;, &#39;distribute&#39;, &#39;having&#39;, &#39;sort&#39;, &#39;qualify&#39;, &#39;offset&#39;, &#39;distinct&#39;, &#39;settings&#39;, &#39;pivots&#39;, &#39;connect&#39;, &#39;limit&#39;, &#39;sample&#39;, &#39;group&#39;, &#39;match&#39;, &#39;format&#39;, &#39;kind&#39;, &#39;into&#39;, &#39;cluster&#39;, &#39;options&#39;, &#39;windows&#39;, &#39;with&#39;, &#39;locks&#39;, &#39;laterals&#39;}</span>
</div>

4
docs/sqlglot/optimizer/simplify.html
View file

@ -3006,7 +3006,7 @@ prefix are statically known.</p>
<div class="attr variable">
<span class="name">DATETRUNC_COMPARISONS</span> =
<input id="DATETRUNC_COMPARISONS-view-value" class="view-value-toggle-state" type="checkbox" aria-hidden="true" tabindex="-1">
<label class="view-value-button pdoc-button" for="DATETRUNC_COMPARISONS-view-value"></label><span class="default_value">{&lt;class &#39;<a href="../expressions.html#EQ">sqlglot.expressions.EQ</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#GT">sqlglot.expressions.GT</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#LTE">sqlglot.expressions.LTE</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#NEQ">sqlglot.expressions.NEQ</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#GTE">sqlglot.expressions.GTE</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#LT">sqlglot.expressions.LT</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#In">sqlglot.expressions.In</a>&#39;&gt;}</span>
<label class="view-value-button pdoc-button" for="DATETRUNC_COMPARISONS-view-value"></label><span class="default_value">{&lt;class &#39;<a href="../expressions.html#LT">sqlglot.expressions.LT</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#GTE">sqlglot.expressions.GTE</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#EQ">sqlglot.expressions.EQ</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#LTE">sqlglot.expressions.LTE</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#NEQ">sqlglot.expressions.NEQ</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#GT">sqlglot.expressions.GT</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#In">sqlglot.expressions.In</a>&#39;&gt;}</span>
</div>
@ -3086,7 +3086,7 @@ prefix are statically known.</p>
<section id="JOINS">
<div class="attr variable">
<span class="name">JOINS</span> =
<span class="default_value">{(&#39;RIGHT&#39;, &#39;&#39;), (&#39;&#39;, &#39;INNER&#39;), (&#39;RIGHT&#39;, &#39;OUTER&#39;), (&#39;&#39;, &#39;&#39;)}</span>
<span class="default_value">{(&#39;&#39;, &#39;INNER&#39;), (&#39;RIGHT&#39;, &#39;OUTER&#39;), (&#39;RIGHT&#39;, &#39;&#39;), (&#39;&#39;, &#39;&#39;)}</span>
</div>

538
docs/sqlglot/optimizer/unnest_subqueries.html
View file

@ -64,7 +64,7 @@
<div class="pdoc-code codehilite"><pre><span></span><span id="L-1"><a href="#L-1"><span class="linenos"> 1</span></a><span class="kn">from</span> <span class="nn">sqlglot</span> <span class="kn">import</span> <span class="n">exp</span>
</span><span id="L-2"><a href="#L-2"><span class="linenos"> 2</span></a><span class="kn">from</span> <span class="nn">sqlglot.helper</span> <span class="kn">import</span> <span class="n">name_sequence</span>
</span><span id="L-3"><a href="#L-3"><span class="linenos"> 3</span></a><span class="kn">from</span> <span class="nn">sqlglot.optimizer.scope</span> <span class="kn">import</span> <span class="n">ScopeType</span><span class="p">,</span> <span class="n">traverse_scope</span>
</span><span id="L-3"><a href="#L-3"><span class="linenos"> 3</span></a><span class="kn">from</span> <span class="nn">sqlglot.optimizer.scope</span> <span class="kn">import</span> <span class="n">ScopeType</span><span class="p">,</span> <span class="n">find_in_scope</span><span class="p">,</span> <span class="n">traverse_scope</span>
</span><span id="L-4"><a href="#L-4"><span class="linenos"> 4</span></a>
</span><span id="L-5"><a href="#L-5"><span class="linenos"> 5</span></a>
</span><span id="L-6"><a href="#L-6"><span class="linenos"> 6</span></a><span class="k">def</span> <span class="nf">unnest_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
@ -128,7 +128,7 @@
</span><span id="L-64"><a href="#L-64"><span class="linenos"> 64</span></a> <span class="p">(</span><span class="ow">not</span> <span class="n">clause</span> <span class="ow">or</span> <span class="n">clause_parent_select</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">parent_select</span><span class="p">)</span>
</span><span id="L-65"><a href="#L-65"><span class="linenos"> 65</span></a> <span class="ow">and</span> <span class="p">(</span>
</span><span id="L-66"><a href="#L-66"><span class="linenos"> 66</span></a> <span class="n">parent_select</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;group&quot;</span><span class="p">)</span>
</span><span id="L-67"><a href="#L-67"><span class="linenos"> 67</span></a> <span class="ow">or</span> <span class="nb">any</span><span class="p">(</span><span class="n">projection</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">)</span> <span class="k">for</span> <span class="n">projection</span> <span class="ow">in</span> <span class="n">parent_select</span><span class="o">.</span><span class="n">selects</span><span class="p">)</span>
</span><span id="L-67"><a href="#L-67"><span class="linenos"> 67</span></a> <span class="ow">or</span> <span class="nb">any</span><span class="p">(</span><span class="n">find_in_scope</span><span class="p">(</span><span class="n">select</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">)</span> <span class="k">for</span> <span class="n">select</span> <span class="ow">in</span> <span class="n">parent_select</span><span class="o">.</span><span class="n">selects</span><span class="p">)</span>
</span><span id="L-68"><a href="#L-68"><span class="linenos"> 68</span></a> <span class="p">)</span>
</span><span id="L-69"><a href="#L-69"><span class="linenos"> 69</span></a> <span class="p">):</span>
</span><span id="L-70"><a href="#L-70"><span class="linenos"> 70</span></a> <span class="n">column</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Max</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">column</span><span class="p">)</span>
@ -165,7 +165,7 @@
</span><span id="L-101"><a href="#L-101"><span class="linenos">101</span></a> <span class="k">if</span> <span class="n">group</span><span class="p">:</span>
</span><span id="L-102"><a href="#L-102"><span class="linenos">102</span></a> <span class="k">if</span> <span class="p">{</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">}</span> <span class="o">!=</span> <span class="nb">set</span><span class="p">(</span><span class="n">group</span><span class="o">.</span><span class="n">expressions</span><span class="p">):</span>
</span><span id="L-103"><a href="#L-103"><span class="linenos">103</span></a> <span class="n">select</span> <span class="o">=</span> <span class="p">(</span>
</span><span id="L-104"><a href="#L-104"><span class="linenos">104</span></a> <span class="n">exp</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="s2">&quot;_q&quot;</span><span class="p">))</span>
</span><span id="L-104"><a href="#L-104"><span class="linenos">104</span></a> <span class="n">exp</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="s2">&quot;_q&quot;</span><span class="p">),</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">))</span>
</span><span id="L-105"><a href="#L-105"><span class="linenos">105</span></a> <span class="o">.</span><span class="n">from_</span><span class="p">(</span><span class="n">select</span><span class="o">.</span><span class="n">subquery</span><span class="p">(</span><span class="s2">&quot;_q&quot;</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-106"><a href="#L-106"><span class="linenos">106</span></a> <span class="o">.</span><span class="n">group_by</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="s2">&quot;_q&quot;</span><span class="p">),</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-107"><a href="#L-107"><span class="linenos">107</span></a> <span class="p">)</span>
@ -216,147 +216,154 @@
</span><span id="L-152"><a href="#L-152"><span class="linenos">152</span></a> <span class="k">return</span>
</span><span id="L-153"><a href="#L-153"><span class="linenos">153</span></a>
</span><span id="L-154"><a href="#L-154"><span class="linenos">154</span></a> <span class="n">is_subquery_projection</span> <span class="o">=</span> <span class="nb">any</span><span class="p">(</span>
</span><span id="L-155"><a href="#L-155"><span class="linenos">155</span></a> <span class="n">node</span> <span class="ow">is</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="n">parent_select</span><span class="o">.</span><span class="n">selects</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">)</span>
</span><span id="L-156"><a href="#L-156"><span class="linenos">156</span></a> <span class="p">)</span>
</span><span id="L-157"><a href="#L-157"><span class="linenos">157</span></a>
</span><span id="L-158"><a href="#L-158"><span class="linenos">158</span></a> <span class="n">value</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">selects</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span id="L-159"><a href="#L-159"><span class="linenos">159</span></a> <span class="n">key_aliases</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-160"><a href="#L-160"><span class="linenos">160</span></a> <span class="n">group_by</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-161"><a href="#L-161"><span class="linenos">161</span></a>
</span><span id="L-162"><a href="#L-162"><span class="linenos">162</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="L-163"><a href="#L-163"><span class="linenos">163</span></a> <span class="c1"># if we filter on the value of the subquery, it needs to be unique</span>
</span><span id="L-164"><a href="#L-164"><span class="linenos">164</span></a> <span class="k">if</span> <span class="n">key</span> <span class="o">==</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">:</span>
</span><span id="L-165"><a href="#L-165"><span class="linenos">165</span></a> <span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span>
</span><span id="L-166"><a href="#L-166"><span class="linenos">166</span></a> <span class="n">group_by</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span><span id="L-167"><a href="#L-167"><span class="linenos">167</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-168"><a href="#L-168"><span class="linenos">168</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">key_aliases</span><span class="p">:</span>
</span><span id="L-169"><a href="#L-169"><span class="linenos">169</span></a> <span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">next_alias_name</span><span class="p">()</span>
</span><span id="L-170"><a href="#L-170"><span class="linenos">170</span></a> <span class="c1"># all predicates that are equalities must also be in the unique</span>
</span><span id="L-171"><a href="#L-171"><span class="linenos">171</span></a> <span class="c1"># so that we don&#39;t do a many to many join</span>
</span><span id="L-172"><a href="#L-172"><span class="linenos">172</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">)</span> <span class="ow">and</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-173"><a href="#L-173"><span class="linenos">173</span></a> <span class="n">group_by</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span><span id="L-174"><a href="#L-174"><span class="linenos">174</span></a>
</span><span id="L-175"><a href="#L-175"><span class="linenos">175</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">find_ancestor</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Predicate</span><span class="p">)</span>
</span><span id="L-155"><a href="#L-155"><span class="linenos">155</span></a> <span class="n">node</span> <span class="ow">is</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-156"><a href="#L-156"><span class="linenos">156</span></a> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">s</span><span class="p">:</span> <span class="n">s</span><span class="o">.</span><span class="n">unalias</span><span class="p">(),</span> <span class="n">parent_select</span><span class="o">.</span><span class="n">selects</span><span class="p">)</span>
</span><span id="L-157"><a href="#L-157"><span class="linenos">157</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">)</span>
</span><span id="L-158"><a href="#L-158"><span class="linenos">158</span></a> <span class="p">)</span>
</span><span id="L-159"><a href="#L-159"><span class="linenos">159</span></a>
</span><span id="L-160"><a href="#L-160"><span class="linenos">160</span></a> <span class="n">value</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">selects</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span id="L-161"><a href="#L-161"><span class="linenos">161</span></a> <span class="n">key_aliases</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-162"><a href="#L-162"><span class="linenos">162</span></a> <span class="n">group_by</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-163"><a href="#L-163"><span class="linenos">163</span></a>
</span><span id="L-164"><a href="#L-164"><span class="linenos">164</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="L-165"><a href="#L-165"><span class="linenos">165</span></a> <span class="c1"># if we filter on the value of the subquery, it needs to be unique</span>
</span><span id="L-166"><a href="#L-166"><span class="linenos">166</span></a> <span class="k">if</span> <span class="n">key</span> <span class="o">==</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">:</span>
</span><span id="L-167"><a href="#L-167"><span class="linenos">167</span></a> <span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span>
</span><span id="L-168"><a href="#L-168"><span class="linenos">168</span></a> <span class="n">group_by</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span><span id="L-169"><a href="#L-169"><span class="linenos">169</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-170"><a href="#L-170"><span class="linenos">170</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">key_aliases</span><span class="p">:</span>
</span><span id="L-171"><a href="#L-171"><span class="linenos">171</span></a> <span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">next_alias_name</span><span class="p">()</span>
</span><span id="L-172"><a href="#L-172"><span class="linenos">172</span></a> <span class="c1"># all predicates that are equalities must also be in the unique</span>
</span><span id="L-173"><a href="#L-173"><span class="linenos">173</span></a> <span class="c1"># so that we don&#39;t do a many to many join</span>
</span><span id="L-174"><a href="#L-174"><span class="linenos">174</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">)</span> <span class="ow">and</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-175"><a href="#L-175"><span class="linenos">175</span></a> <span class="n">group_by</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span><span id="L-176"><a href="#L-176"><span class="linenos">176</span></a>
</span><span id="L-177"><a href="#L-177"><span class="linenos">177</span></a> <span class="c1"># if the value of the subquery is not an agg or a key, we need to collect it into an array</span>
</span><span id="L-178"><a href="#L-178"><span class="linenos">178</span></a> <span class="c1"># so that it can be grouped. For subquery projections, we use a MAX aggregation instead.</span>
</span><span id="L-179"><a href="#L-179"><span class="linenos">179</span></a> <span class="n">agg_func</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Max</span> <span class="k">if</span> <span class="n">is_subquery_projection</span> <span class="k">else</span> <span class="n">exp</span><span class="o">.</span><span class="n">ArrayAgg</span>
</span><span id="L-180"><a href="#L-180"><span class="linenos">180</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">value</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">)</span> <span class="ow">and</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-181"><a href="#L-181"><span class="linenos">181</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
</span><span id="L-182"><a href="#L-182"><span class="linenos">182</span></a> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">agg_func</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">),</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="n">quoted</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span><span id="L-183"><a href="#L-183"><span class="linenos">183</span></a> <span class="n">append</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="L-184"><a href="#L-184"><span class="linenos">184</span></a> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="L-185"><a href="#L-185"><span class="linenos">185</span></a> <span class="p">)</span>
</span><span id="L-186"><a href="#L-186"><span class="linenos">186</span></a>
</span><span id="L-187"><a href="#L-187"><span class="linenos">187</span></a> <span class="c1"># exists queries should not have any selects as it only checks if there are any rows</span>
</span><span id="L-188"><a href="#L-188"><span class="linenos">188</span></a> <span class="c1"># all selects will be added by the optimizer and only used for join keys</span>
</span><span id="L-189"><a href="#L-189"><span class="linenos">189</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">):</span>
</span><span id="L-190"><a href="#L-190"><span class="linenos">190</span></a> <span class="n">select</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="s2">&quot;expressions&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-191"><a href="#L-191"><span class="linenos">191</span></a>
</span><span id="L-192"><a href="#L-192"><span class="linenos">192</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">alias</span> <span class="ow">in</span> <span class="n">key_aliases</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="L-193"><a href="#L-193"><span class="linenos">193</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-194"><a href="#L-194"><span class="linenos">194</span></a> <span class="c1"># add all keys to the projections of the subquery</span>
</span><span id="L-195"><a href="#L-195"><span class="linenos">195</span></a> <span class="c1"># so that we can use it as a join key</span>
</span><span id="L-196"><a href="#L-196"><span class="linenos">196</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">)</span> <span class="ow">or</span> <span class="n">key</span> <span class="o">!=</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">:</span>
</span><span id="L-197"><a href="#L-197"><span class="linenos">197</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2"> AS </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-198"><a href="#L-198"><span class="linenos">198</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-199"><a href="#L-199"><span class="linenos">199</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">agg_func</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">key</span><span class="o">.</span><span class="n">copy</span><span class="p">()),</span> <span class="n">alias</span><span class="p">,</span> <span class="n">quoted</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-200"><a href="#L-200"><span class="linenos">200</span></a>
</span><span id="L-201"><a href="#L-201"><span class="linenos">201</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="L-202"><a href="#L-202"><span class="linenos">202</span></a> <span class="n">other</span> <span class="o">=</span> <span class="n">_other_operand</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">)</span>
</span><span id="L-203"><a href="#L-203"><span class="linenos">203</span></a>
</span><span id="L-204"><a href="#L-204"><span class="linenos">204</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">):</span>
</span><span id="L-205"><a href="#L-205"><span class="linenos">205</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">key_aliases</span><span class="o">.</span><span class="n">values</span><span class="p">())[</span><span class="mi">0</span><span class="p">],</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="L-206"><a href="#L-206"><span class="linenos">206</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;NOT </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2"> IS NULL&quot;</span><span class="p">)</span>
</span><span id="L-207"><a href="#L-207"><span class="linenos">207</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">All</span><span class="p">):</span>
</span><span id="L-208"><a href="#L-208"><span class="linenos">208</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="L-209"><a href="#L-209"><span class="linenos">209</span></a> <span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;ARRAY_ALL(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; _x = </span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2">)&quot;</span>
</span><span id="L-210"><a href="#L-210"><span class="linenos">210</span></a> <span class="p">)</span>
</span><span id="L-211"><a href="#L-211"><span class="linenos">211</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Any</span><span class="p">):</span>
</span><span id="L-212"><a href="#L-212"><span class="linenos">212</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-213"><a href="#L-213"><span class="linenos">213</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="L-214"><a href="#L-214"><span class="linenos">214</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-215"><a href="#L-215"><span class="linenos">215</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;ARRAY_ANY(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; _x = </span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2">)&quot;</span><span class="p">)</span>
</span><span id="L-216"><a href="#L-216"><span class="linenos">216</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">In</span><span class="p">):</span>
</span><span id="L-217"><a href="#L-217"><span class="linenos">217</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-218"><a href="#L-218"><span class="linenos">218</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="L-219"><a href="#L-219"><span class="linenos">219</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-220"><a href="#L-220"><span class="linenos">220</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="L-221"><a href="#L-221"><span class="linenos">221</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="L-222"><a href="#L-222"><span class="linenos">222</span></a> <span class="sa">f</span><span class="s2">&quot;ARRAY_ANY(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; _x = </span><span class="si">{</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">this</span><span class="si">}</span><span class="s2">)&quot;</span><span class="p">,</span>
</span><span id="L-223"><a href="#L-223"><span class="linenos">223</span></a> <span class="p">)</span>
</span><span id="L-224"><a href="#L-224"><span class="linenos">224</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-225"><a href="#L-225"><span class="linenos">225</span></a> <span class="k">if</span> <span class="n">is_subquery_projection</span><span class="p">:</span>
</span><span id="L-226"><a href="#L-226"><span class="linenos">226</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">alias</span><span class="p">,</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">)</span>
</span><span id="L-227"><a href="#L-227"><span class="linenos">227</span></a>
</span><span id="L-228"><a href="#L-228"><span class="linenos">228</span></a> <span class="c1"># COUNT always returns 0 on empty datasets, so we need take that into consideration here</span>
</span><span id="L-229"><a href="#L-229"><span class="linenos">229</span></a> <span class="c1"># by transforming all counts into 0 and using that as the coalesced value</span>
</span><span id="L-230"><a href="#L-230"><span class="linenos">230</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Count</span><span class="p">):</span>
</span><span id="L-231"><a href="#L-231"><span class="linenos">231</span></a>
</span><span id="L-232"><a href="#L-232"><span class="linenos">232</span></a> <span class="k">def</span> <span class="nf">remove_aggs</span><span class="p">(</span><span class="n">node</span><span class="p">):</span>
</span><span id="L-233"><a href="#L-233"><span class="linenos">233</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Count</span><span class="p">):</span>
</span><span id="L-234"><a href="#L-234"><span class="linenos">234</span></a> <span class="k">return</span> <span class="n">exp</span><span class="o">.</span><span class="n">Literal</span><span class="o">.</span><span class="n">number</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span id="L-235"><a href="#L-235"><span class="linenos">235</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">):</span>
</span><span id="L-236"><a href="#L-236"><span class="linenos">236</span></a> <span class="k">return</span> <span class="n">exp</span><span class="o">.</span><span class="n">null</span><span class="p">()</span>
</span><span id="L-237"><a href="#L-237"><span class="linenos">237</span></a> <span class="k">return</span> <span class="n">node</span>
</span><span id="L-238"><a href="#L-238"><span class="linenos">238</span></a>
</span><span id="L-239"><a href="#L-239"><span class="linenos">239</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Coalesce</span><span class="p">(</span>
</span><span id="L-240"><a href="#L-240"><span class="linenos">240</span></a> <span class="n">this</span><span class="o">=</span><span class="n">alias</span><span class="p">,</span>
</span><span id="L-241"><a href="#L-241"><span class="linenos">241</span></a> <span class="n">expressions</span><span class="o">=</span><span class="p">[</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">remove_aggs</span><span class="p">)],</span>
</span><span id="L-242"><a href="#L-242"><span class="linenos">242</span></a> <span class="p">)</span>
</span><span id="L-243"><a href="#L-243"><span class="linenos">243</span></a>
</span><span id="L-244"><a href="#L-244"><span class="linenos">244</span></a> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">alias</span><span class="p">)</span>
</span><span id="L-245"><a href="#L-245"><span class="linenos">245</span></a>
</span><span id="L-246"><a href="#L-246"><span class="linenos">246</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">column</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="L-247"><a href="#L-247"><span class="linenos">247</span></a> <span class="n">predicate</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">true</span><span class="p">())</span>
</span><span id="L-248"><a href="#L-248"><span class="linenos">248</span></a> <span class="n">nested</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="L-249"><a href="#L-249"><span class="linenos">249</span></a>
</span><span id="L-250"><a href="#L-250"><span class="linenos">250</span></a> <span class="k">if</span> <span class="n">is_subquery_projection</span><span class="p">:</span>
</span><span id="L-251"><a href="#L-251"><span class="linenos">251</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
</span><span id="L-252"><a href="#L-252"><span class="linenos">252</span></a> <span class="k">continue</span>
</span><span id="L-253"><a href="#L-253"><span class="linenos">253</span></a>
</span><span id="L-254"><a href="#L-254"><span class="linenos">254</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-255"><a href="#L-255"><span class="linenos">255</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
</span><span id="L-256"><a href="#L-256"><span class="linenos">256</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">):</span>
</span><span id="L-257"><a href="#L-257"><span class="linenos">257</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="L-258"><a href="#L-258"><span class="linenos">258</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="L-259"><a href="#L-259"><span class="linenos">259</span></a> <span class="sa">f</span><span class="s2">&quot;(</span><span class="si">{</span><span class="n">parent_predicate</span><span class="si">}</span><span class="s2"> AND ARRAY_CONTAINS(</span><span class="si">{</span><span class="n">nested</span><span class="si">}</span><span class="s2">, </span><span class="si">{</span><span class="n">column</span><span class="si">}</span><span class="s2">))&quot;</span><span class="p">,</span>
</span><span id="L-260"><a href="#L-260"><span class="linenos">260</span></a> <span class="p">)</span>
</span><span id="L-261"><a href="#L-261"><span class="linenos">261</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-262"><a href="#L-262"><span class="linenos">262</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">to_identifier</span><span class="p">(</span><span class="s2">&quot;_x&quot;</span><span class="p">))</span>
</span><span id="L-263"><a href="#L-263"><span class="linenos">263</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="L-264"><a href="#L-264"><span class="linenos">264</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="L-265"><a href="#L-265"><span class="linenos">265</span></a> <span class="sa">f</span><span class="s2">&quot;(</span><span class="si">{</span><span class="n">parent_predicate</span><span class="si">}</span><span class="s2"> AND ARRAY_ANY(</span><span class="si">{</span><span class="n">nested</span><span class="si">}</span><span class="s2">, _x -&gt; </span><span class="si">{</span><span class="n">predicate</span><span class="si">}</span><span class="s2">))&quot;</span><span class="p">,</span>
</span><span id="L-266"><a href="#L-266"><span class="linenos">266</span></a> <span class="p">)</span>
</span><span id="L-267"><a href="#L-267"><span class="linenos">267</span></a>
</span><span id="L-268"><a href="#L-268"><span class="linenos">268</span></a> <span class="n">parent_select</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
</span><span id="L-269"><a href="#L-269"><span class="linenos">269</span></a> <span class="n">select</span><span class="o">.</span><span class="n">group_by</span><span class="p">(</span><span class="o">*</span><span class="n">group_by</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span><span id="L-270"><a href="#L-270"><span class="linenos">270</span></a> <span class="n">on</span><span class="o">=</span><span class="p">[</span><span class="n">predicate</span> <span class="k">for</span> <span class="o">*</span><span class="n">_</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">)],</span>
</span><span id="L-271"><a href="#L-271"><span class="linenos">271</span></a> <span class="n">join_type</span><span class="o">=</span><span class="s2">&quot;LEFT&quot;</span><span class="p">,</span>
</span><span id="L-272"><a href="#L-272"><span class="linenos">272</span></a> <span class="n">join_alias</span><span class="o">=</span><span class="n">table_alias</span><span class="p">,</span>
</span><span id="L-273"><a href="#L-273"><span class="linenos">273</span></a> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="L-274"><a href="#L-274"><span class="linenos">274</span></a> <span class="p">)</span>
</span><span id="L-275"><a href="#L-275"><span class="linenos">275</span></a>
</span><span id="L-276"><a href="#L-276"><span class="linenos">276</span></a>
</span><span id="L-277"><a href="#L-277"><span class="linenos">277</span></a><span class="k">def</span> <span class="nf">_replace</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">condition</span><span class="p">):</span>
</span><span id="L-278"><a href="#L-278"><span class="linenos">278</span></a> <span class="k">return</span> <span class="n">expression</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">condition</span><span class="p">(</span><span class="n">condition</span><span class="p">))</span>
</span><span id="L-279"><a href="#L-279"><span class="linenos">279</span></a>
</span><span id="L-280"><a href="#L-280"><span class="linenos">280</span></a>
</span><span id="L-281"><a href="#L-281"><span class="linenos">281</span></a><span class="k">def</span> <span class="nf">_other_operand</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
</span><span id="L-282"><a href="#L-282"><span class="linenos">282</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">In</span><span class="p">):</span>
</span><span id="L-283"><a href="#L-283"><span class="linenos">283</span></a> <span class="k">return</span> <span class="n">expression</span><span class="o">.</span><span class="n">this</span>
</span><span id="L-284"><a href="#L-284"><span class="linenos">284</span></a>
</span><span id="L-285"><a href="#L-285"><span class="linenos">285</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Any</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">All</span><span class="p">)):</span>
</span><span id="L-286"><a href="#L-286"><span class="linenos">286</span></a> <span class="k">return</span> <span class="n">_other_operand</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="p">)</span>
</span><span id="L-177"><a href="#L-177"><span class="linenos">177</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">find_ancestor</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Predicate</span><span class="p">)</span>
</span><span id="L-178"><a href="#L-178"><span class="linenos">178</span></a>
</span><span id="L-179"><a href="#L-179"><span class="linenos">179</span></a> <span class="c1"># if the value of the subquery is not an agg or a key, we need to collect it into an array</span>
</span><span id="L-180"><a href="#L-180"><span class="linenos">180</span></a> <span class="c1"># so that it can be grouped. For subquery projections, we use a MAX aggregation instead.</span>
</span><span id="L-181"><a href="#L-181"><span class="linenos">181</span></a> <span class="n">agg_func</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Max</span> <span class="k">if</span> <span class="n">is_subquery_projection</span> <span class="k">else</span> <span class="n">exp</span><span class="o">.</span><span class="n">ArrayAgg</span>
</span><span id="L-182"><a href="#L-182"><span class="linenos">182</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">value</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">)</span> <span class="ow">and</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-183"><a href="#L-183"><span class="linenos">183</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
</span><span id="L-184"><a href="#L-184"><span class="linenos">184</span></a> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">agg_func</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">),</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="n">quoted</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span><span id="L-185"><a href="#L-185"><span class="linenos">185</span></a> <span class="n">append</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="L-186"><a href="#L-186"><span class="linenos">186</span></a> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="L-187"><a href="#L-187"><span class="linenos">187</span></a> <span class="p">)</span>
</span><span id="L-188"><a href="#L-188"><span class="linenos">188</span></a>
</span><span id="L-189"><a href="#L-189"><span class="linenos">189</span></a> <span class="c1"># exists queries should not have any selects as it only checks if there are any rows</span>
</span><span id="L-190"><a href="#L-190"><span class="linenos">190</span></a> <span class="c1"># all selects will be added by the optimizer and only used for join keys</span>
</span><span id="L-191"><a href="#L-191"><span class="linenos">191</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">):</span>
</span><span id="L-192"><a href="#L-192"><span class="linenos">192</span></a> <span class="n">select</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="s2">&quot;expressions&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-193"><a href="#L-193"><span class="linenos">193</span></a>
</span><span id="L-194"><a href="#L-194"><span class="linenos">194</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">alias</span> <span class="ow">in</span> <span class="n">key_aliases</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="L-195"><a href="#L-195"><span class="linenos">195</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-196"><a href="#L-196"><span class="linenos">196</span></a> <span class="c1"># add all keys to the projections of the subquery</span>
</span><span id="L-197"><a href="#L-197"><span class="linenos">197</span></a> <span class="c1"># so that we can use it as a join key</span>
</span><span id="L-198"><a href="#L-198"><span class="linenos">198</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">)</span> <span class="ow">or</span> <span class="n">key</span> <span class="o">!=</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">:</span>
</span><span id="L-199"><a href="#L-199"><span class="linenos">199</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2"> AS </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-200"><a href="#L-200"><span class="linenos">200</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-201"><a href="#L-201"><span class="linenos">201</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">agg_func</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">key</span><span class="o">.</span><span class="n">copy</span><span class="p">()),</span> <span class="n">alias</span><span class="p">,</span> <span class="n">quoted</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-202"><a href="#L-202"><span class="linenos">202</span></a>
</span><span id="L-203"><a href="#L-203"><span class="linenos">203</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="L-204"><a href="#L-204"><span class="linenos">204</span></a> <span class="n">other</span> <span class="o">=</span> <span class="n">_other_operand</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">)</span>
</span><span id="L-205"><a href="#L-205"><span class="linenos">205</span></a> <span class="n">op_type</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">)</span> <span class="k">if</span> <span class="n">parent_predicate</span> <span class="k">else</span> <span class="kc">None</span>
</span><span id="L-206"><a href="#L-206"><span class="linenos">206</span></a>
</span><span id="L-207"><a href="#L-207"><span class="linenos">207</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">):</span>
</span><span id="L-208"><a href="#L-208"><span class="linenos">208</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">key_aliases</span><span class="o">.</span><span class="n">values</span><span class="p">())[</span><span class="mi">0</span><span class="p">],</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="L-209"><a href="#L-209"><span class="linenos">209</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;NOT </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2"> IS NULL&quot;</span><span class="p">)</span>
</span><span id="L-210"><a href="#L-210"><span class="linenos">210</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">All</span><span class="p">):</span>
</span><span id="L-211"><a href="#L-211"><span class="linenos">211</span></a> <span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">op_type</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Binary</span><span class="p">)</span>
</span><span id="L-212"><a href="#L-212"><span class="linenos">212</span></a> <span class="n">predicate</span> <span class="o">=</span> <span class="n">op_type</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">other</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="s2">&quot;_x&quot;</span><span class="p">))</span>
</span><span id="L-213"><a href="#L-213"><span class="linenos">213</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="L-214"><a href="#L-214"><span class="linenos">214</span></a> <span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;ARRAY_ALL(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; </span><span class="si">{</span><span class="n">predicate</span><span class="si">}</span><span class="s2">)&quot;</span>
</span><span id="L-215"><a href="#L-215"><span class="linenos">215</span></a> <span class="p">)</span>
</span><span id="L-216"><a href="#L-216"><span class="linenos">216</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Any</span><span class="p">):</span>
</span><span id="L-217"><a href="#L-217"><span class="linenos">217</span></a> <span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">op_type</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Binary</span><span class="p">)</span>
</span><span id="L-218"><a href="#L-218"><span class="linenos">218</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-219"><a href="#L-219"><span class="linenos">219</span></a> <span class="n">predicate</span> <span class="o">=</span> <span class="n">op_type</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">other</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">alias</span><span class="p">)</span>
</span><span id="L-220"><a href="#L-220"><span class="linenos">220</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="n">predicate</span><span class="p">)</span>
</span><span id="L-221"><a href="#L-221"><span class="linenos">221</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-222"><a href="#L-222"><span class="linenos">222</span></a> <span class="n">predicate</span> <span class="o">=</span> <span class="n">op_type</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">other</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="s2">&quot;_x&quot;</span><span class="p">))</span>
</span><span id="L-223"><a href="#L-223"><span class="linenos">223</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;ARRAY_ANY(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; </span><span class="si">{</span><span class="n">predicate</span><span class="si">}</span><span class="s2">)&quot;</span><span class="p">)</span>
</span><span id="L-224"><a href="#L-224"><span class="linenos">224</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">In</span><span class="p">):</span>
</span><span id="L-225"><a href="#L-225"><span class="linenos">225</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-226"><a href="#L-226"><span class="linenos">226</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="L-227"><a href="#L-227"><span class="linenos">227</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-228"><a href="#L-228"><span class="linenos">228</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="L-229"><a href="#L-229"><span class="linenos">229</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="L-230"><a href="#L-230"><span class="linenos">230</span></a> <span class="sa">f</span><span class="s2">&quot;ARRAY_ANY(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; _x = </span><span class="si">{</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">this</span><span class="si">}</span><span class="s2">)&quot;</span><span class="p">,</span>
</span><span id="L-231"><a href="#L-231"><span class="linenos">231</span></a> <span class="p">)</span>
</span><span id="L-232"><a href="#L-232"><span class="linenos">232</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-233"><a href="#L-233"><span class="linenos">233</span></a> <span class="k">if</span> <span class="n">is_subquery_projection</span> <span class="ow">and</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">:</span>
</span><span id="L-234"><a href="#L-234"><span class="linenos">234</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">alias</span><span class="p">,</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">)</span>
</span><span id="L-235"><a href="#L-235"><span class="linenos">235</span></a>
</span><span id="L-236"><a href="#L-236"><span class="linenos">236</span></a> <span class="c1"># COUNT always returns 0 on empty datasets, so we need take that into consideration here</span>
</span><span id="L-237"><a href="#L-237"><span class="linenos">237</span></a> <span class="c1"># by transforming all counts into 0 and using that as the coalesced value</span>
</span><span id="L-238"><a href="#L-238"><span class="linenos">238</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Count</span><span class="p">):</span>
</span><span id="L-239"><a href="#L-239"><span class="linenos">239</span></a>
</span><span id="L-240"><a href="#L-240"><span class="linenos">240</span></a> <span class="k">def</span> <span class="nf">remove_aggs</span><span class="p">(</span><span class="n">node</span><span class="p">):</span>
</span><span id="L-241"><a href="#L-241"><span class="linenos">241</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Count</span><span class="p">):</span>
</span><span id="L-242"><a href="#L-242"><span class="linenos">242</span></a> <span class="k">return</span> <span class="n">exp</span><span class="o">.</span><span class="n">Literal</span><span class="o">.</span><span class="n">number</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span id="L-243"><a href="#L-243"><span class="linenos">243</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">):</span>
</span><span id="L-244"><a href="#L-244"><span class="linenos">244</span></a> <span class="k">return</span> <span class="n">exp</span><span class="o">.</span><span class="n">null</span><span class="p">()</span>
</span><span id="L-245"><a href="#L-245"><span class="linenos">245</span></a> <span class="k">return</span> <span class="n">node</span>
</span><span id="L-246"><a href="#L-246"><span class="linenos">246</span></a>
</span><span id="L-247"><a href="#L-247"><span class="linenos">247</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Coalesce</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">alias</span><span class="p">,</span> <span class="n">expressions</span><span class="o">=</span><span class="p">[</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">remove_aggs</span><span class="p">)])</span>
</span><span id="L-248"><a href="#L-248"><span class="linenos">248</span></a>
</span><span id="L-249"><a href="#L-249"><span class="linenos">249</span></a> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">alias</span><span class="p">)</span>
</span><span id="L-250"><a href="#L-250"><span class="linenos">250</span></a>
</span><span id="L-251"><a href="#L-251"><span class="linenos">251</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">column</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="L-252"><a href="#L-252"><span class="linenos">252</span></a> <span class="n">predicate</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">true</span><span class="p">())</span>
</span><span id="L-253"><a href="#L-253"><span class="linenos">253</span></a> <span class="n">nested</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="L-254"><a href="#L-254"><span class="linenos">254</span></a>
</span><span id="L-255"><a href="#L-255"><span class="linenos">255</span></a> <span class="k">if</span> <span class="n">is_subquery_projection</span><span class="p">:</span>
</span><span id="L-256"><a href="#L-256"><span class="linenos">256</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
</span><span id="L-257"><a href="#L-257"><span class="linenos">257</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">):</span>
</span><span id="L-258"><a href="#L-258"><span class="linenos">258</span></a> <span class="n">parent_select</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-259"><a href="#L-259"><span class="linenos">259</span></a> <span class="k">continue</span>
</span><span id="L-260"><a href="#L-260"><span class="linenos">260</span></a>
</span><span id="L-261"><a href="#L-261"><span class="linenos">261</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="L-262"><a href="#L-262"><span class="linenos">262</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
</span><span id="L-263"><a href="#L-263"><span class="linenos">263</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">):</span>
</span><span id="L-264"><a href="#L-264"><span class="linenos">264</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="L-265"><a href="#L-265"><span class="linenos">265</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="L-266"><a href="#L-266"><span class="linenos">266</span></a> <span class="sa">f</span><span class="s2">&quot;(</span><span class="si">{</span><span class="n">parent_predicate</span><span class="si">}</span><span class="s2"> AND ARRAY_CONTAINS(</span><span class="si">{</span><span class="n">nested</span><span class="si">}</span><span class="s2">, </span><span class="si">{</span><span class="n">column</span><span class="si">}</span><span class="s2">))&quot;</span><span class="p">,</span>
</span><span id="L-267"><a href="#L-267"><span class="linenos">267</span></a> <span class="p">)</span>
</span><span id="L-268"><a href="#L-268"><span class="linenos">268</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-269"><a href="#L-269"><span class="linenos">269</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">to_identifier</span><span class="p">(</span><span class="s2">&quot;_x&quot;</span><span class="p">))</span>
</span><span id="L-270"><a href="#L-270"><span class="linenos">270</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="L-271"><a href="#L-271"><span class="linenos">271</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="L-272"><a href="#L-272"><span class="linenos">272</span></a> <span class="sa">f</span><span class="s2">&quot;(</span><span class="si">{</span><span class="n">parent_predicate</span><span class="si">}</span><span class="s2"> AND ARRAY_ANY(</span><span class="si">{</span><span class="n">nested</span><span class="si">}</span><span class="s2">, _x -&gt; </span><span class="si">{</span><span class="n">predicate</span><span class="si">}</span><span class="s2">))&quot;</span><span class="p">,</span>
</span><span id="L-273"><a href="#L-273"><span class="linenos">273</span></a> <span class="p">)</span>
</span><span id="L-274"><a href="#L-274"><span class="linenos">274</span></a>
</span><span id="L-275"><a href="#L-275"><span class="linenos">275</span></a> <span class="n">parent_select</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
</span><span id="L-276"><a href="#L-276"><span class="linenos">276</span></a> <span class="n">select</span><span class="o">.</span><span class="n">group_by</span><span class="p">(</span><span class="o">*</span><span class="n">group_by</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span><span id="L-277"><a href="#L-277"><span class="linenos">277</span></a> <span class="n">on</span><span class="o">=</span><span class="p">[</span><span class="n">predicate</span> <span class="k">for</span> <span class="o">*</span><span class="n">_</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">)],</span>
</span><span id="L-278"><a href="#L-278"><span class="linenos">278</span></a> <span class="n">join_type</span><span class="o">=</span><span class="s2">&quot;LEFT&quot;</span><span class="p">,</span>
</span><span id="L-279"><a href="#L-279"><span class="linenos">279</span></a> <span class="n">join_alias</span><span class="o">=</span><span class="n">table_alias</span><span class="p">,</span>
</span><span id="L-280"><a href="#L-280"><span class="linenos">280</span></a> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="L-281"><a href="#L-281"><span class="linenos">281</span></a> <span class="p">)</span>
</span><span id="L-282"><a href="#L-282"><span class="linenos">282</span></a>
</span><span id="L-283"><a href="#L-283"><span class="linenos">283</span></a>
</span><span id="L-284"><a href="#L-284"><span class="linenos">284</span></a><span class="k">def</span> <span class="nf">_replace</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">condition</span><span class="p">):</span>
</span><span id="L-285"><a href="#L-285"><span class="linenos">285</span></a> <span class="k">return</span> <span class="n">expression</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">condition</span><span class="p">(</span><span class="n">condition</span><span class="p">))</span>
</span><span id="L-286"><a href="#L-286"><span class="linenos">286</span></a>
</span><span id="L-287"><a href="#L-287"><span class="linenos">287</span></a>
</span><span id="L-288"><a href="#L-288"><span class="linenos">288</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Binary</span><span class="p">):</span>
</span><span id="L-289"><a href="#L-289"><span class="linenos">289</span></a> <span class="k">return</span> <span class="p">(</span>
</span><span id="L-290"><a href="#L-290"><span class="linenos">290</span></a> <span class="n">expression</span><span class="o">.</span><span class="n">right</span>
</span><span id="L-291"><a href="#L-291"><span class="linenos">291</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">left</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Any</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">All</span><span class="p">))</span>
</span><span id="L-292"><a href="#L-292"><span class="linenos">292</span></a> <span class="k">else</span> <span class="n">expression</span><span class="o">.</span><span class="n">left</span>
</span><span id="L-293"><a href="#L-293"><span class="linenos">293</span></a> <span class="p">)</span>
</span><span id="L-288"><a href="#L-288"><span class="linenos">288</span></a><span class="k">def</span> <span class="nf">_other_operand</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
</span><span id="L-289"><a href="#L-289"><span class="linenos">289</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">In</span><span class="p">):</span>
</span><span id="L-290"><a href="#L-290"><span class="linenos">290</span></a> <span class="k">return</span> <span class="n">expression</span><span class="o">.</span><span class="n">this</span>
</span><span id="L-291"><a href="#L-291"><span class="linenos">291</span></a>
</span><span id="L-292"><a href="#L-292"><span class="linenos">292</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Any</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">All</span><span class="p">)):</span>
</span><span id="L-293"><a href="#L-293"><span class="linenos">293</span></a> <span class="k">return</span> <span class="n">_other_operand</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="p">)</span>
</span><span id="L-294"><a href="#L-294"><span class="linenos">294</span></a>
</span><span id="L-295"><a href="#L-295"><span class="linenos">295</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="L-295"><a href="#L-295"><span class="linenos">295</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Binary</span><span class="p">):</span>
</span><span id="L-296"><a href="#L-296"><span class="linenos">296</span></a> <span class="k">return</span> <span class="p">(</span>
</span><span id="L-297"><a href="#L-297"><span class="linenos">297</span></a> <span class="n">expression</span><span class="o">.</span><span class="n">right</span>
</span><span id="L-298"><a href="#L-298"><span class="linenos">298</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">left</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Any</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">All</span><span class="p">))</span>
</span><span id="L-299"><a href="#L-299"><span class="linenos">299</span></a> <span class="k">else</span> <span class="n">expression</span><span class="o">.</span><span class="n">left</span>
</span><span id="L-300"><a href="#L-300"><span class="linenos">300</span></a> <span class="p">)</span>
</span><span id="L-301"><a href="#L-301"><span class="linenos">301</span></a>
</span><span id="L-302"><a href="#L-302"><span class="linenos">302</span></a> <span class="k">return</span> <span class="kc">None</span>
</span></pre></div>
@ -477,7 +484,7 @@ Convert correlated or vectorized subqueries into a group by so it is not a many
</span><span id="unnest-65"><a href="#unnest-65"><span class="linenos"> 65</span></a> <span class="p">(</span><span class="ow">not</span> <span class="n">clause</span> <span class="ow">or</span> <span class="n">clause_parent_select</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">parent_select</span><span class="p">)</span>
</span><span id="unnest-66"><a href="#unnest-66"><span class="linenos"> 66</span></a> <span class="ow">and</span> <span class="p">(</span>
</span><span id="unnest-67"><a href="#unnest-67"><span class="linenos"> 67</span></a> <span class="n">parent_select</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;group&quot;</span><span class="p">)</span>
</span><span id="unnest-68"><a href="#unnest-68"><span class="linenos"> 68</span></a> <span class="ow">or</span> <span class="nb">any</span><span class="p">(</span><span class="n">projection</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">)</span> <span class="k">for</span> <span class="n">projection</span> <span class="ow">in</span> <span class="n">parent_select</span><span class="o">.</span><span class="n">selects</span><span class="p">)</span>
</span><span id="unnest-68"><a href="#unnest-68"><span class="linenos"> 68</span></a> <span class="ow">or</span> <span class="nb">any</span><span class="p">(</span><span class="n">find_in_scope</span><span class="p">(</span><span class="n">select</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">)</span> <span class="k">for</span> <span class="n">select</span> <span class="ow">in</span> <span class="n">parent_select</span><span class="o">.</span><span class="n">selects</span><span class="p">)</span>
</span><span id="unnest-69"><a href="#unnest-69"><span class="linenos"> 69</span></a> <span class="p">)</span>
</span><span id="unnest-70"><a href="#unnest-70"><span class="linenos"> 70</span></a> <span class="p">):</span>
</span><span id="unnest-71"><a href="#unnest-71"><span class="linenos"> 71</span></a> <span class="n">column</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Max</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">column</span><span class="p">)</span>
@ -514,7 +521,7 @@ Convert correlated or vectorized subqueries into a group by so it is not a many
</span><span id="unnest-102"><a href="#unnest-102"><span class="linenos">102</span></a> <span class="k">if</span> <span class="n">group</span><span class="p">:</span>
</span><span id="unnest-103"><a href="#unnest-103"><span class="linenos">103</span></a> <span class="k">if</span> <span class="p">{</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">}</span> <span class="o">!=</span> <span class="nb">set</span><span class="p">(</span><span class="n">group</span><span class="o">.</span><span class="n">expressions</span><span class="p">):</span>
</span><span id="unnest-104"><a href="#unnest-104"><span class="linenos">104</span></a> <span class="n">select</span> <span class="o">=</span> <span class="p">(</span>
</span><span id="unnest-105"><a href="#unnest-105"><span class="linenos">105</span></a> <span class="n">exp</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="s2">&quot;_q&quot;</span><span class="p">))</span>
</span><span id="unnest-105"><a href="#unnest-105"><span class="linenos">105</span></a> <span class="n">exp</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="s2">&quot;_q&quot;</span><span class="p">),</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">))</span>
</span><span id="unnest-106"><a href="#unnest-106"><span class="linenos">106</span></a> <span class="o">.</span><span class="n">from_</span><span class="p">(</span><span class="n">select</span><span class="o">.</span><span class="n">subquery</span><span class="p">(</span><span class="s2">&quot;_q&quot;</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="unnest-107"><a href="#unnest-107"><span class="linenos">107</span></a> <span class="o">.</span><span class="n">group_by</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="s2">&quot;_q&quot;</span><span class="p">),</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="unnest-108"><a href="#unnest-108"><span class="linenos">108</span></a> <span class="p">)</span>
@ -580,126 +587,133 @@ Convert correlated or vectorized subqueries into a group by so it is not a many
</span><span id="decorrelate-153"><a href="#decorrelate-153"><span class="linenos">153</span></a> <span class="k">return</span>
</span><span id="decorrelate-154"><a href="#decorrelate-154"><span class="linenos">154</span></a>
</span><span id="decorrelate-155"><a href="#decorrelate-155"><span class="linenos">155</span></a> <span class="n">is_subquery_projection</span> <span class="o">=</span> <span class="nb">any</span><span class="p">(</span>
</span><span id="decorrelate-156"><a href="#decorrelate-156"><span class="linenos">156</span></a> <span class="n">node</span> <span class="ow">is</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="n">parent_select</span><span class="o">.</span><span class="n">selects</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">)</span>
</span><span id="decorrelate-157"><a href="#decorrelate-157"><span class="linenos">157</span></a> <span class="p">)</span>
</span><span id="decorrelate-158"><a href="#decorrelate-158"><span class="linenos">158</span></a>
</span><span id="decorrelate-159"><a href="#decorrelate-159"><span class="linenos">159</span></a> <span class="n">value</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">selects</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span id="decorrelate-160"><a href="#decorrelate-160"><span class="linenos">160</span></a> <span class="n">key_aliases</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="decorrelate-161"><a href="#decorrelate-161"><span class="linenos">161</span></a> <span class="n">group_by</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="decorrelate-162"><a href="#decorrelate-162"><span class="linenos">162</span></a>
</span><span id="decorrelate-163"><a href="#decorrelate-163"><span class="linenos">163</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="decorrelate-164"><a href="#decorrelate-164"><span class="linenos">164</span></a> <span class="c1"># if we filter on the value of the subquery, it needs to be unique</span>
</span><span id="decorrelate-165"><a href="#decorrelate-165"><span class="linenos">165</span></a> <span class="k">if</span> <span class="n">key</span> <span class="o">==</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">:</span>
</span><span id="decorrelate-166"><a href="#decorrelate-166"><span class="linenos">166</span></a> <span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span>
</span><span id="decorrelate-167"><a href="#decorrelate-167"><span class="linenos">167</span></a> <span class="n">group_by</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span><span id="decorrelate-168"><a href="#decorrelate-168"><span class="linenos">168</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-169"><a href="#decorrelate-169"><span class="linenos">169</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">key_aliases</span><span class="p">:</span>
</span><span id="decorrelate-170"><a href="#decorrelate-170"><span class="linenos">170</span></a> <span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">next_alias_name</span><span class="p">()</span>
</span><span id="decorrelate-171"><a href="#decorrelate-171"><span class="linenos">171</span></a> <span class="c1"># all predicates that are equalities must also be in the unique</span>
</span><span id="decorrelate-172"><a href="#decorrelate-172"><span class="linenos">172</span></a> <span class="c1"># so that we don&#39;t do a many to many join</span>
</span><span id="decorrelate-173"><a href="#decorrelate-173"><span class="linenos">173</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">)</span> <span class="ow">and</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-174"><a href="#decorrelate-174"><span class="linenos">174</span></a> <span class="n">group_by</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span><span id="decorrelate-175"><a href="#decorrelate-175"><span class="linenos">175</span></a>
</span><span id="decorrelate-176"><a href="#decorrelate-176"><span class="linenos">176</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">find_ancestor</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Predicate</span><span class="p">)</span>
</span><span id="decorrelate-156"><a href="#decorrelate-156"><span class="linenos">156</span></a> <span class="n">node</span> <span class="ow">is</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span>
</span><span id="decorrelate-157"><a href="#decorrelate-157"><span class="linenos">157</span></a> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">s</span><span class="p">:</span> <span class="n">s</span><span class="o">.</span><span class="n">unalias</span><span class="p">(),</span> <span class="n">parent_select</span><span class="o">.</span><span class="n">selects</span><span class="p">)</span>
</span><span id="decorrelate-158"><a href="#decorrelate-158"><span class="linenos">158</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">)</span>
</span><span id="decorrelate-159"><a href="#decorrelate-159"><span class="linenos">159</span></a> <span class="p">)</span>
</span><span id="decorrelate-160"><a href="#decorrelate-160"><span class="linenos">160</span></a>
</span><span id="decorrelate-161"><a href="#decorrelate-161"><span class="linenos">161</span></a> <span class="n">value</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">selects</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span id="decorrelate-162"><a href="#decorrelate-162"><span class="linenos">162</span></a> <span class="n">key_aliases</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="decorrelate-163"><a href="#decorrelate-163"><span class="linenos">163</span></a> <span class="n">group_by</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="decorrelate-164"><a href="#decorrelate-164"><span class="linenos">164</span></a>
</span><span id="decorrelate-165"><a href="#decorrelate-165"><span class="linenos">165</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="decorrelate-166"><a href="#decorrelate-166"><span class="linenos">166</span></a> <span class="c1"># if we filter on the value of the subquery, it needs to be unique</span>
</span><span id="decorrelate-167"><a href="#decorrelate-167"><span class="linenos">167</span></a> <span class="k">if</span> <span class="n">key</span> <span class="o">==</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">:</span>
</span><span id="decorrelate-168"><a href="#decorrelate-168"><span class="linenos">168</span></a> <span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span>
</span><span id="decorrelate-169"><a href="#decorrelate-169"><span class="linenos">169</span></a> <span class="n">group_by</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span><span id="decorrelate-170"><a href="#decorrelate-170"><span class="linenos">170</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-171"><a href="#decorrelate-171"><span class="linenos">171</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">key_aliases</span><span class="p">:</span>
</span><span id="decorrelate-172"><a href="#decorrelate-172"><span class="linenos">172</span></a> <span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">next_alias_name</span><span class="p">()</span>
</span><span id="decorrelate-173"><a href="#decorrelate-173"><span class="linenos">173</span></a> <span class="c1"># all predicates that are equalities must also be in the unique</span>
</span><span id="decorrelate-174"><a href="#decorrelate-174"><span class="linenos">174</span></a> <span class="c1"># so that we don&#39;t do a many to many join</span>
</span><span id="decorrelate-175"><a href="#decorrelate-175"><span class="linenos">175</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">)</span> <span class="ow">and</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-176"><a href="#decorrelate-176"><span class="linenos">176</span></a> <span class="n">group_by</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span><span id="decorrelate-177"><a href="#decorrelate-177"><span class="linenos">177</span></a>
</span><span id="decorrelate-178"><a href="#decorrelate-178"><span class="linenos">178</span></a> <span class="c1"># if the value of the subquery is not an agg or a key, we need to collect it into an array</span>
</span><span id="decorrelate-179"><a href="#decorrelate-179"><span class="linenos">179</span></a> <span class="c1"># so that it can be grouped. For subquery projections, we use a MAX aggregation instead.</span>
</span><span id="decorrelate-180"><a href="#decorrelate-180"><span class="linenos">180</span></a> <span class="n">agg_func</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Max</span> <span class="k">if</span> <span class="n">is_subquery_projection</span> <span class="k">else</span> <span class="n">exp</span><span class="o">.</span><span class="n">ArrayAgg</span>
</span><span id="decorrelate-181"><a href="#decorrelate-181"><span class="linenos">181</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">value</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">)</span> <span class="ow">and</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-182"><a href="#decorrelate-182"><span class="linenos">182</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
</span><span id="decorrelate-183"><a href="#decorrelate-183"><span class="linenos">183</span></a> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">agg_func</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">),</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="n">quoted</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span><span id="decorrelate-184"><a href="#decorrelate-184"><span class="linenos">184</span></a> <span class="n">append</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="decorrelate-185"><a href="#decorrelate-185"><span class="linenos">185</span></a> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="decorrelate-186"><a href="#decorrelate-186"><span class="linenos">186</span></a> <span class="p">)</span>
</span><span id="decorrelate-187"><a href="#decorrelate-187"><span class="linenos">187</span></a>
</span><span id="decorrelate-188"><a href="#decorrelate-188"><span class="linenos">188</span></a> <span class="c1"># exists queries should not have any selects as it only checks if there are any rows</span>
</span><span id="decorrelate-189"><a href="#decorrelate-189"><span class="linenos">189</span></a> <span class="c1"># all selects will be added by the optimizer and only used for join keys</span>
</span><span id="decorrelate-190"><a href="#decorrelate-190"><span class="linenos">190</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">):</span>
</span><span id="decorrelate-191"><a href="#decorrelate-191"><span class="linenos">191</span></a> <span class="n">select</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="s2">&quot;expressions&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="decorrelate-192"><a href="#decorrelate-192"><span class="linenos">192</span></a>
</span><span id="decorrelate-193"><a href="#decorrelate-193"><span class="linenos">193</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">alias</span> <span class="ow">in</span> <span class="n">key_aliases</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="decorrelate-194"><a href="#decorrelate-194"><span class="linenos">194</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-195"><a href="#decorrelate-195"><span class="linenos">195</span></a> <span class="c1"># add all keys to the projections of the subquery</span>
</span><span id="decorrelate-196"><a href="#decorrelate-196"><span class="linenos">196</span></a> <span class="c1"># so that we can use it as a join key</span>
</span><span id="decorrelate-197"><a href="#decorrelate-197"><span class="linenos">197</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">)</span> <span class="ow">or</span> <span class="n">key</span> <span class="o">!=</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">:</span>
</span><span id="decorrelate-198"><a href="#decorrelate-198"><span class="linenos">198</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2"> AS </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="decorrelate-199"><a href="#decorrelate-199"><span class="linenos">199</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-200"><a href="#decorrelate-200"><span class="linenos">200</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">agg_func</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">key</span><span class="o">.</span><span class="n">copy</span><span class="p">()),</span> <span class="n">alias</span><span class="p">,</span> <span class="n">quoted</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="decorrelate-201"><a href="#decorrelate-201"><span class="linenos">201</span></a>
</span><span id="decorrelate-202"><a href="#decorrelate-202"><span class="linenos">202</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="decorrelate-203"><a href="#decorrelate-203"><span class="linenos">203</span></a> <span class="n">other</span> <span class="o">=</span> <span class="n">_other_operand</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">)</span>
</span><span id="decorrelate-204"><a href="#decorrelate-204"><span class="linenos">204</span></a>
</span><span id="decorrelate-205"><a href="#decorrelate-205"><span class="linenos">205</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">):</span>
</span><span id="decorrelate-206"><a href="#decorrelate-206"><span class="linenos">206</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">key_aliases</span><span class="o">.</span><span class="n">values</span><span class="p">())[</span><span class="mi">0</span><span class="p">],</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="decorrelate-207"><a href="#decorrelate-207"><span class="linenos">207</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;NOT </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2"> IS NULL&quot;</span><span class="p">)</span>
</span><span id="decorrelate-208"><a href="#decorrelate-208"><span class="linenos">208</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">All</span><span class="p">):</span>
</span><span id="decorrelate-209"><a href="#decorrelate-209"><span class="linenos">209</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="decorrelate-210"><a href="#decorrelate-210"><span class="linenos">210</span></a> <span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;ARRAY_ALL(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; _x = </span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2">)&quot;</span>
</span><span id="decorrelate-211"><a href="#decorrelate-211"><span class="linenos">211</span></a> <span class="p">)</span>
</span><span id="decorrelate-212"><a href="#decorrelate-212"><span class="linenos">212</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Any</span><span class="p">):</span>
</span><span id="decorrelate-213"><a href="#decorrelate-213"><span class="linenos">213</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-214"><a href="#decorrelate-214"><span class="linenos">214</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="decorrelate-215"><a href="#decorrelate-215"><span class="linenos">215</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-216"><a href="#decorrelate-216"><span class="linenos">216</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;ARRAY_ANY(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; _x = </span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2">)&quot;</span><span class="p">)</span>
</span><span id="decorrelate-217"><a href="#decorrelate-217"><span class="linenos">217</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">In</span><span class="p">):</span>
</span><span id="decorrelate-218"><a href="#decorrelate-218"><span class="linenos">218</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-219"><a href="#decorrelate-219"><span class="linenos">219</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="decorrelate-220"><a href="#decorrelate-220"><span class="linenos">220</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-221"><a href="#decorrelate-221"><span class="linenos">221</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="decorrelate-222"><a href="#decorrelate-222"><span class="linenos">222</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="decorrelate-223"><a href="#decorrelate-223"><span class="linenos">223</span></a> <span class="sa">f</span><span class="s2">&quot;ARRAY_ANY(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; _x = </span><span class="si">{</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">this</span><span class="si">}</span><span class="s2">)&quot;</span><span class="p">,</span>
</span><span id="decorrelate-224"><a href="#decorrelate-224"><span class="linenos">224</span></a> <span class="p">)</span>
</span><span id="decorrelate-225"><a href="#decorrelate-225"><span class="linenos">225</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-226"><a href="#decorrelate-226"><span class="linenos">226</span></a> <span class="k">if</span> <span class="n">is_subquery_projection</span><span class="p">:</span>
</span><span id="decorrelate-227"><a href="#decorrelate-227"><span class="linenos">227</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">alias</span><span class="p">,</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">)</span>
</span><span id="decorrelate-228"><a href="#decorrelate-228"><span class="linenos">228</span></a>
</span><span id="decorrelate-229"><a href="#decorrelate-229"><span class="linenos">229</span></a> <span class="c1"># COUNT always returns 0 on empty datasets, so we need take that into consideration here</span>
</span><span id="decorrelate-230"><a href="#decorrelate-230"><span class="linenos">230</span></a> <span class="c1"># by transforming all counts into 0 and using that as the coalesced value</span>
</span><span id="decorrelate-231"><a href="#decorrelate-231"><span class="linenos">231</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Count</span><span class="p">):</span>
</span><span id="decorrelate-232"><a href="#decorrelate-232"><span class="linenos">232</span></a>
</span><span id="decorrelate-233"><a href="#decorrelate-233"><span class="linenos">233</span></a> <span class="k">def</span> <span class="nf">remove_aggs</span><span class="p">(</span><span class="n">node</span><span class="p">):</span>
</span><span id="decorrelate-234"><a href="#decorrelate-234"><span class="linenos">234</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Count</span><span class="p">):</span>
</span><span id="decorrelate-235"><a href="#decorrelate-235"><span class="linenos">235</span></a> <span class="k">return</span> <span class="n">exp</span><span class="o">.</span><span class="n">Literal</span><span class="o">.</span><span class="n">number</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span id="decorrelate-236"><a href="#decorrelate-236"><span class="linenos">236</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">):</span>
</span><span id="decorrelate-237"><a href="#decorrelate-237"><span class="linenos">237</span></a> <span class="k">return</span> <span class="n">exp</span><span class="o">.</span><span class="n">null</span><span class="p">()</span>
</span><span id="decorrelate-238"><a href="#decorrelate-238"><span class="linenos">238</span></a> <span class="k">return</span> <span class="n">node</span>
</span><span id="decorrelate-239"><a href="#decorrelate-239"><span class="linenos">239</span></a>
</span><span id="decorrelate-240"><a href="#decorrelate-240"><span class="linenos">240</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Coalesce</span><span class="p">(</span>
</span><span id="decorrelate-241"><a href="#decorrelate-241"><span class="linenos">241</span></a> <span class="n">this</span><span class="o">=</span><span class="n">alias</span><span class="p">,</span>
</span><span id="decorrelate-242"><a href="#decorrelate-242"><span class="linenos">242</span></a> <span class="n">expressions</span><span class="o">=</span><span class="p">[</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">remove_aggs</span><span class="p">)],</span>
</span><span id="decorrelate-243"><a href="#decorrelate-243"><span class="linenos">243</span></a> <span class="p">)</span>
</span><span id="decorrelate-244"><a href="#decorrelate-244"><span class="linenos">244</span></a>
</span><span id="decorrelate-245"><a href="#decorrelate-245"><span class="linenos">245</span></a> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">alias</span><span class="p">)</span>
</span><span id="decorrelate-246"><a href="#decorrelate-246"><span class="linenos">246</span></a>
</span><span id="decorrelate-247"><a href="#decorrelate-247"><span class="linenos">247</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">column</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="decorrelate-248"><a href="#decorrelate-248"><span class="linenos">248</span></a> <span class="n">predicate</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">true</span><span class="p">())</span>
</span><span id="decorrelate-249"><a href="#decorrelate-249"><span class="linenos">249</span></a> <span class="n">nested</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="decorrelate-250"><a href="#decorrelate-250"><span class="linenos">250</span></a>
</span><span id="decorrelate-251"><a href="#decorrelate-251"><span class="linenos">251</span></a> <span class="k">if</span> <span class="n">is_subquery_projection</span><span class="p">:</span>
</span><span id="decorrelate-252"><a href="#decorrelate-252"><span class="linenos">252</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
</span><span id="decorrelate-253"><a href="#decorrelate-253"><span class="linenos">253</span></a> <span class="k">continue</span>
</span><span id="decorrelate-254"><a href="#decorrelate-254"><span class="linenos">254</span></a>
</span><span id="decorrelate-255"><a href="#decorrelate-255"><span class="linenos">255</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-256"><a href="#decorrelate-256"><span class="linenos">256</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
</span><span id="decorrelate-257"><a href="#decorrelate-257"><span class="linenos">257</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">):</span>
</span><span id="decorrelate-258"><a href="#decorrelate-258"><span class="linenos">258</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="decorrelate-259"><a href="#decorrelate-259"><span class="linenos">259</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="decorrelate-260"><a href="#decorrelate-260"><span class="linenos">260</span></a> <span class="sa">f</span><span class="s2">&quot;(</span><span class="si">{</span><span class="n">parent_predicate</span><span class="si">}</span><span class="s2"> AND ARRAY_CONTAINS(</span><span class="si">{</span><span class="n">nested</span><span class="si">}</span><span class="s2">, </span><span class="si">{</span><span class="n">column</span><span class="si">}</span><span class="s2">))&quot;</span><span class="p">,</span>
</span><span id="decorrelate-261"><a href="#decorrelate-261"><span class="linenos">261</span></a> <span class="p">)</span>
</span><span id="decorrelate-262"><a href="#decorrelate-262"><span class="linenos">262</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-263"><a href="#decorrelate-263"><span class="linenos">263</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">to_identifier</span><span class="p">(</span><span class="s2">&quot;_x&quot;</span><span class="p">))</span>
</span><span id="decorrelate-264"><a href="#decorrelate-264"><span class="linenos">264</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="decorrelate-265"><a href="#decorrelate-265"><span class="linenos">265</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="decorrelate-266"><a href="#decorrelate-266"><span class="linenos">266</span></a> <span class="sa">f</span><span class="s2">&quot;(</span><span class="si">{</span><span class="n">parent_predicate</span><span class="si">}</span><span class="s2"> AND ARRAY_ANY(</span><span class="si">{</span><span class="n">nested</span><span class="si">}</span><span class="s2">, _x -&gt; </span><span class="si">{</span><span class="n">predicate</span><span class="si">}</span><span class="s2">))&quot;</span><span class="p">,</span>
</span><span id="decorrelate-267"><a href="#decorrelate-267"><span class="linenos">267</span></a> <span class="p">)</span>
</span><span id="decorrelate-268"><a href="#decorrelate-268"><span class="linenos">268</span></a>
</span><span id="decorrelate-269"><a href="#decorrelate-269"><span class="linenos">269</span></a> <span class="n">parent_select</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
</span><span id="decorrelate-270"><a href="#decorrelate-270"><span class="linenos">270</span></a> <span class="n">select</span><span class="o">.</span><span class="n">group_by</span><span class="p">(</span><span class="o">*</span><span class="n">group_by</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span><span id="decorrelate-271"><a href="#decorrelate-271"><span class="linenos">271</span></a> <span class="n">on</span><span class="o">=</span><span class="p">[</span><span class="n">predicate</span> <span class="k">for</span> <span class="o">*</span><span class="n">_</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">)],</span>
</span><span id="decorrelate-272"><a href="#decorrelate-272"><span class="linenos">272</span></a> <span class="n">join_type</span><span class="o">=</span><span class="s2">&quot;LEFT&quot;</span><span class="p">,</span>
</span><span id="decorrelate-273"><a href="#decorrelate-273"><span class="linenos">273</span></a> <span class="n">join_alias</span><span class="o">=</span><span class="n">table_alias</span><span class="p">,</span>
</span><span id="decorrelate-274"><a href="#decorrelate-274"><span class="linenos">274</span></a> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="decorrelate-275"><a href="#decorrelate-275"><span class="linenos">275</span></a> <span class="p">)</span>
</span><span id="decorrelate-178"><a href="#decorrelate-178"><span class="linenos">178</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">find_ancestor</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Predicate</span><span class="p">)</span>
</span><span id="decorrelate-179"><a href="#decorrelate-179"><span class="linenos">179</span></a>
</span><span id="decorrelate-180"><a href="#decorrelate-180"><span class="linenos">180</span></a> <span class="c1"># if the value of the subquery is not an agg or a key, we need to collect it into an array</span>
</span><span id="decorrelate-181"><a href="#decorrelate-181"><span class="linenos">181</span></a> <span class="c1"># so that it can be grouped. For subquery projections, we use a MAX aggregation instead.</span>
</span><span id="decorrelate-182"><a href="#decorrelate-182"><span class="linenos">182</span></a> <span class="n">agg_func</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Max</span> <span class="k">if</span> <span class="n">is_subquery_projection</span> <span class="k">else</span> <span class="n">exp</span><span class="o">.</span><span class="n">ArrayAgg</span>
</span><span id="decorrelate-183"><a href="#decorrelate-183"><span class="linenos">183</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">value</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">)</span> <span class="ow">and</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-184"><a href="#decorrelate-184"><span class="linenos">184</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
</span><span id="decorrelate-185"><a href="#decorrelate-185"><span class="linenos">185</span></a> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">agg_func</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">),</span> <span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="n">quoted</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span><span id="decorrelate-186"><a href="#decorrelate-186"><span class="linenos">186</span></a> <span class="n">append</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="decorrelate-187"><a href="#decorrelate-187"><span class="linenos">187</span></a> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="decorrelate-188"><a href="#decorrelate-188"><span class="linenos">188</span></a> <span class="p">)</span>
</span><span id="decorrelate-189"><a href="#decorrelate-189"><span class="linenos">189</span></a>
</span><span id="decorrelate-190"><a href="#decorrelate-190"><span class="linenos">190</span></a> <span class="c1"># exists queries should not have any selects as it only checks if there are any rows</span>
</span><span id="decorrelate-191"><a href="#decorrelate-191"><span class="linenos">191</span></a> <span class="c1"># all selects will be added by the optimizer and only used for join keys</span>
</span><span id="decorrelate-192"><a href="#decorrelate-192"><span class="linenos">192</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">):</span>
</span><span id="decorrelate-193"><a href="#decorrelate-193"><span class="linenos">193</span></a> <span class="n">select</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="s2">&quot;expressions&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="decorrelate-194"><a href="#decorrelate-194"><span class="linenos">194</span></a>
</span><span id="decorrelate-195"><a href="#decorrelate-195"><span class="linenos">195</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">alias</span> <span class="ow">in</span> <span class="n">key_aliases</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="decorrelate-196"><a href="#decorrelate-196"><span class="linenos">196</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-197"><a href="#decorrelate-197"><span class="linenos">197</span></a> <span class="c1"># add all keys to the projections of the subquery</span>
</span><span id="decorrelate-198"><a href="#decorrelate-198"><span class="linenos">198</span></a> <span class="c1"># so that we can use it as a join key</span>
</span><span id="decorrelate-199"><a href="#decorrelate-199"><span class="linenos">199</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">)</span> <span class="ow">or</span> <span class="n">key</span> <span class="o">!=</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="p">:</span>
</span><span id="decorrelate-200"><a href="#decorrelate-200"><span class="linenos">200</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2"> AS </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="decorrelate-201"><a href="#decorrelate-201"><span class="linenos">201</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-202"><a href="#decorrelate-202"><span class="linenos">202</span></a> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">agg_func</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">key</span><span class="o">.</span><span class="n">copy</span><span class="p">()),</span> <span class="n">alias</span><span class="p">,</span> <span class="n">quoted</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="decorrelate-203"><a href="#decorrelate-203"><span class="linenos">203</span></a>
</span><span id="decorrelate-204"><a href="#decorrelate-204"><span class="linenos">204</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">value</span><span class="o">.</span><span class="n">alias</span><span class="p">,</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="decorrelate-205"><a href="#decorrelate-205"><span class="linenos">205</span></a> <span class="n">other</span> <span class="o">=</span> <span class="n">_other_operand</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">)</span>
</span><span id="decorrelate-206"><a href="#decorrelate-206"><span class="linenos">206</span></a> <span class="n">op_type</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">)</span> <span class="k">if</span> <span class="n">parent_predicate</span> <span class="k">else</span> <span class="kc">None</span>
</span><span id="decorrelate-207"><a href="#decorrelate-207"><span class="linenos">207</span></a>
</span><span id="decorrelate-208"><a href="#decorrelate-208"><span class="linenos">208</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Exists</span><span class="p">):</span>
</span><span id="decorrelate-209"><a href="#decorrelate-209"><span class="linenos">209</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">key_aliases</span><span class="o">.</span><span class="n">values</span><span class="p">())[</span><span class="mi">0</span><span class="p">],</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="decorrelate-210"><a href="#decorrelate-210"><span class="linenos">210</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;NOT </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2"> IS NULL&quot;</span><span class="p">)</span>
</span><span id="decorrelate-211"><a href="#decorrelate-211"><span class="linenos">211</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">All</span><span class="p">):</span>
</span><span id="decorrelate-212"><a href="#decorrelate-212"><span class="linenos">212</span></a> <span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">op_type</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Binary</span><span class="p">)</span>
</span><span id="decorrelate-213"><a href="#decorrelate-213"><span class="linenos">213</span></a> <span class="n">predicate</span> <span class="o">=</span> <span class="n">op_type</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">other</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="s2">&quot;_x&quot;</span><span class="p">))</span>
</span><span id="decorrelate-214"><a href="#decorrelate-214"><span class="linenos">214</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="decorrelate-215"><a href="#decorrelate-215"><span class="linenos">215</span></a> <span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;ARRAY_ALL(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; </span><span class="si">{</span><span class="n">predicate</span><span class="si">}</span><span class="s2">)&quot;</span>
</span><span id="decorrelate-216"><a href="#decorrelate-216"><span class="linenos">216</span></a> <span class="p">)</span>
</span><span id="decorrelate-217"><a href="#decorrelate-217"><span class="linenos">217</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Any</span><span class="p">):</span>
</span><span id="decorrelate-218"><a href="#decorrelate-218"><span class="linenos">218</span></a> <span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">op_type</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Binary</span><span class="p">)</span>
</span><span id="decorrelate-219"><a href="#decorrelate-219"><span class="linenos">219</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-220"><a href="#decorrelate-220"><span class="linenos">220</span></a> <span class="n">predicate</span> <span class="o">=</span> <span class="n">op_type</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">other</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">alias</span><span class="p">)</span>
</span><span id="decorrelate-221"><a href="#decorrelate-221"><span class="linenos">221</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="n">predicate</span><span class="p">)</span>
</span><span id="decorrelate-222"><a href="#decorrelate-222"><span class="linenos">222</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-223"><a href="#decorrelate-223"><span class="linenos">223</span></a> <span class="n">predicate</span> <span class="o">=</span> <span class="n">op_type</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">other</span><span class="p">,</span> <span class="n">expression</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="s2">&quot;_x&quot;</span><span class="p">))</span>
</span><span id="decorrelate-224"><a href="#decorrelate-224"><span class="linenos">224</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;ARRAY_ANY(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; </span><span class="si">{</span><span class="n">predicate</span><span class="si">}</span><span class="s2">)&quot;</span><span class="p">)</span>
</span><span id="decorrelate-225"><a href="#decorrelate-225"><span class="linenos">225</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">In</span><span class="p">):</span>
</span><span id="decorrelate-226"><a href="#decorrelate-226"><span class="linenos">226</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">this</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-227"><a href="#decorrelate-227"><span class="linenos">227</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span><span class="n">parent_predicate</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">other</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="decorrelate-228"><a href="#decorrelate-228"><span class="linenos">228</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-229"><a href="#decorrelate-229"><span class="linenos">229</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="decorrelate-230"><a href="#decorrelate-230"><span class="linenos">230</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="decorrelate-231"><a href="#decorrelate-231"><span class="linenos">231</span></a> <span class="sa">f</span><span class="s2">&quot;ARRAY_ANY(</span><span class="si">{</span><span class="n">alias</span><span class="si">}</span><span class="s2">, _x -&gt; _x = </span><span class="si">{</span><span class="n">parent_predicate</span><span class="o">.</span><span class="n">this</span><span class="si">}</span><span class="s2">)&quot;</span><span class="p">,</span>
</span><span id="decorrelate-232"><a href="#decorrelate-232"><span class="linenos">232</span></a> <span class="p">)</span>
</span><span id="decorrelate-233"><a href="#decorrelate-233"><span class="linenos">233</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-234"><a href="#decorrelate-234"><span class="linenos">234</span></a> <span class="k">if</span> <span class="n">is_subquery_projection</span> <span class="ow">and</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">:</span>
</span><span id="decorrelate-235"><a href="#decorrelate-235"><span class="linenos">235</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">alias</span><span class="p">,</span> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">)</span>
</span><span id="decorrelate-236"><a href="#decorrelate-236"><span class="linenos">236</span></a>
</span><span id="decorrelate-237"><a href="#decorrelate-237"><span class="linenos">237</span></a> <span class="c1"># COUNT always returns 0 on empty datasets, so we need take that into consideration here</span>
</span><span id="decorrelate-238"><a href="#decorrelate-238"><span class="linenos">238</span></a> <span class="c1"># by transforming all counts into 0 and using that as the coalesced value</span>
</span><span id="decorrelate-239"><a href="#decorrelate-239"><span class="linenos">239</span></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Count</span><span class="p">):</span>
</span><span id="decorrelate-240"><a href="#decorrelate-240"><span class="linenos">240</span></a>
</span><span id="decorrelate-241"><a href="#decorrelate-241"><span class="linenos">241</span></a> <span class="k">def</span> <span class="nf">remove_aggs</span><span class="p">(</span><span class="n">node</span><span class="p">):</span>
</span><span id="decorrelate-242"><a href="#decorrelate-242"><span class="linenos">242</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Count</span><span class="p">):</span>
</span><span id="decorrelate-243"><a href="#decorrelate-243"><span class="linenos">243</span></a> <span class="k">return</span> <span class="n">exp</span><span class="o">.</span><span class="n">Literal</span><span class="o">.</span><span class="n">number</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span id="decorrelate-244"><a href="#decorrelate-244"><span class="linenos">244</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">AggFunc</span><span class="p">):</span>
</span><span id="decorrelate-245"><a href="#decorrelate-245"><span class="linenos">245</span></a> <span class="k">return</span> <span class="n">exp</span><span class="o">.</span><span class="n">null</span><span class="p">()</span>
</span><span id="decorrelate-246"><a href="#decorrelate-246"><span class="linenos">246</span></a> <span class="k">return</span> <span class="n">node</span>
</span><span id="decorrelate-247"><a href="#decorrelate-247"><span class="linenos">247</span></a>
</span><span id="decorrelate-248"><a href="#decorrelate-248"><span class="linenos">248</span></a> <span class="n">alias</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">Coalesce</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">alias</span><span class="p">,</span> <span class="n">expressions</span><span class="o">=</span><span class="p">[</span><span class="n">value</span><span class="o">.</span><span class="n">this</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">remove_aggs</span><span class="p">)])</span>
</span><span id="decorrelate-249"><a href="#decorrelate-249"><span class="linenos">249</span></a>
</span><span id="decorrelate-250"><a href="#decorrelate-250"><span class="linenos">250</span></a> <span class="n">select</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">alias</span><span class="p">)</span>
</span><span id="decorrelate-251"><a href="#decorrelate-251"><span class="linenos">251</span></a>
</span><span id="decorrelate-252"><a href="#decorrelate-252"><span class="linenos">252</span></a> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">column</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="decorrelate-253"><a href="#decorrelate-253"><span class="linenos">253</span></a> <span class="n">predicate</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">true</span><span class="p">())</span>
</span><span id="decorrelate-254"><a href="#decorrelate-254"><span class="linenos">254</span></a> <span class="n">nested</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="n">key_aliases</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="n">table_alias</span><span class="p">)</span>
</span><span id="decorrelate-255"><a href="#decorrelate-255"><span class="linenos">255</span></a>
</span><span id="decorrelate-256"><a href="#decorrelate-256"><span class="linenos">256</span></a> <span class="k">if</span> <span class="n">is_subquery_projection</span><span class="p">:</span>
</span><span id="decorrelate-257"><a href="#decorrelate-257"><span class="linenos">257</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
</span><span id="decorrelate-258"><a href="#decorrelate-258"><span class="linenos">258</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">):</span>
</span><span id="decorrelate-259"><a href="#decorrelate-259"><span class="linenos">259</span></a> <span class="n">parent_select</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="decorrelate-260"><a href="#decorrelate-260"><span class="linenos">260</span></a> <span class="k">continue</span>
</span><span id="decorrelate-261"><a href="#decorrelate-261"><span class="linenos">261</span></a>
</span><span id="decorrelate-262"><a href="#decorrelate-262"><span class="linenos">262</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">group_by</span><span class="p">:</span>
</span><span id="decorrelate-263"><a href="#decorrelate-263"><span class="linenos">263</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
</span><span id="decorrelate-264"><a href="#decorrelate-264"><span class="linenos">264</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">):</span>
</span><span id="decorrelate-265"><a href="#decorrelate-265"><span class="linenos">265</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="decorrelate-266"><a href="#decorrelate-266"><span class="linenos">266</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="decorrelate-267"><a href="#decorrelate-267"><span class="linenos">267</span></a> <span class="sa">f</span><span class="s2">&quot;(</span><span class="si">{</span><span class="n">parent_predicate</span><span class="si">}</span><span class="s2"> AND ARRAY_CONTAINS(</span><span class="si">{</span><span class="n">nested</span><span class="si">}</span><span class="s2">, </span><span class="si">{</span><span class="n">column</span><span class="si">}</span><span class="s2">))&quot;</span><span class="p">,</span>
</span><span id="decorrelate-268"><a href="#decorrelate-268"><span class="linenos">268</span></a> <span class="p">)</span>
</span><span id="decorrelate-269"><a href="#decorrelate-269"><span class="linenos">269</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="decorrelate-270"><a href="#decorrelate-270"><span class="linenos">270</span></a> <span class="n">key</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">to_identifier</span><span class="p">(</span><span class="s2">&quot;_x&quot;</span><span class="p">))</span>
</span><span id="decorrelate-271"><a href="#decorrelate-271"><span class="linenos">271</span></a> <span class="n">parent_predicate</span> <span class="o">=</span> <span class="n">_replace</span><span class="p">(</span>
</span><span id="decorrelate-272"><a href="#decorrelate-272"><span class="linenos">272</span></a> <span class="n">parent_predicate</span><span class="p">,</span>
</span><span id="decorrelate-273"><a href="#decorrelate-273"><span class="linenos">273</span></a> <span class="sa">f</span><span class="s2">&quot;(</span><span class="si">{</span><span class="n">parent_predicate</span><span class="si">}</span><span class="s2"> AND ARRAY_ANY(</span><span class="si">{</span><span class="n">nested</span><span class="si">}</span><span class="s2">, _x -&gt; </span><span class="si">{</span><span class="n">predicate</span><span class="si">}</span><span class="s2">))&quot;</span><span class="p">,</span>
</span><span id="decorrelate-274"><a href="#decorrelate-274"><span class="linenos">274</span></a> <span class="p">)</span>
</span><span id="decorrelate-275"><a href="#decorrelate-275"><span class="linenos">275</span></a>
</span><span id="decorrelate-276"><a href="#decorrelate-276"><span class="linenos">276</span></a> <span class="n">parent_select</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
</span><span id="decorrelate-277"><a href="#decorrelate-277"><span class="linenos">277</span></a> <span class="n">select</span><span class="o">.</span><span class="n">group_by</span><span class="p">(</span><span class="o">*</span><span class="n">group_by</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span><span id="decorrelate-278"><a href="#decorrelate-278"><span class="linenos">278</span></a> <span class="n">on</span><span class="o">=</span><span class="p">[</span><span class="n">predicate</span> <span class="k">for</span> <span class="o">*</span><span class="n">_</span><span class="p">,</span> <span class="n">predicate</span> <span class="ow">in</span> <span class="n">keys</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">predicate</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">EQ</span><span class="p">)],</span>
</span><span id="decorrelate-279"><a href="#decorrelate-279"><span class="linenos">279</span></a> <span class="n">join_type</span><span class="o">=</span><span class="s2">&quot;LEFT&quot;</span><span class="p">,</span>
</span><span id="decorrelate-280"><a href="#decorrelate-280"><span class="linenos">280</span></a> <span class="n">join_alias</span><span class="o">=</span><span class="n">table_alias</span><span class="p">,</span>
</span><span id="decorrelate-281"><a href="#decorrelate-281"><span class="linenos">281</span></a> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span><span id="decorrelate-282"><a href="#decorrelate-282"><span class="linenos">282</span></a> <span class="p">)</span>
</span></pre></div>

25142
docs/sqlglot/parser.html
View file

File diff suppressed because one or more lines are too long

818
docs/sqlglot/schema.html
View file

@ -178,7 +178,7 @@
</span><span id="L-6"><a href="#L-6"><span class="linenos"> 6</span></a><span class="kn">from</span> <span class="nn">sqlglot</span> <span class="kn">import</span> <span class="n">expressions</span> <span class="k">as</span> <span class="n">exp</span>
</span><span id="L-7"><a href="#L-7"><span class="linenos"> 7</span></a><span class="kn">from</span> <span class="nn">sqlglot.dialects.dialect</span> <span class="kn">import</span> <span class="n">Dialect</span>
</span><span id="L-8"><a href="#L-8"><span class="linenos"> 8</span></a><span class="kn">from</span> <span class="nn">sqlglot.errors</span> <span class="kn">import</span> <span class="n">SchemaError</span>
</span><span id="L-9"><a href="#L-9"><span class="linenos"> 9</span></a><span class="kn">from</span> <span class="nn">sqlglot.helper</span> <span class="kn">import</span> <span class="n">dict_depth</span>
</span><span id="L-9"><a href="#L-9"><span class="linenos"> 9</span></a><span class="kn">from</span> <span class="nn">sqlglot.helper</span> <span class="kn">import</span> <span class="n">dict_depth</span><span class="p">,</span> <span class="n">first</span>
</span><span id="L-10"><a href="#L-10"><span class="linenos"> 10</span></a><span class="kn">from</span> <span class="nn">sqlglot.trie</span> <span class="kn">import</span> <span class="n">TrieResult</span><span class="p">,</span> <span class="n">in_trie</span><span class="p">,</span> <span class="n">new_trie</span>
</span><span id="L-11"><a href="#L-11"><span class="linenos"> 11</span></a>
</span><span id="L-12"><a href="#L-12"><span class="linenos"> 12</span></a><span class="k">if</span> <span class="n">t</span><span class="o">.</span><span class="n">TYPE_CHECKING</span><span class="p">:</span>
@ -346,7 +346,7 @@
</span><span id="L-174"><a href="#L-174"><span class="linenos">174</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="L-175"><a href="#L-175"><span class="linenos">175</span></a>
</span><span id="L-176"><a href="#L-176"><span class="linenos">176</span></a> <span class="k">if</span> <span class="n">value</span> <span class="o">==</span> <span class="n">TrieResult</span><span class="o">.</span><span class="n">PREFIX</span><span class="p">:</span>
</span><span id="L-177"><a href="#L-177"><span class="linenos">177</span></a> <span class="n">possibilities</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">trie</span><span class="p">,</span> <span class="n">depth</span><span class="o">=</span><span class="n">dict_depth</span><span class="p">(</span><span class="n">trie</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
</span><span id="L-177"><a href="#L-177"><span class="linenos">177</span></a> <span class="n">possibilities</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">trie</span><span class="p">)</span>
</span><span id="L-178"><a href="#L-178"><span class="linenos">178</span></a>
</span><span id="L-179"><a href="#L-179"><span class="linenos">179</span></a> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">possibilities</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="L-180"><a href="#L-180"><span class="linenos">180</span></a> <span class="n">parts</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">possibilities</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
@ -534,213 +534,219 @@
</span><span id="L-362"><a href="#L-362"><span class="linenos">362</span></a><span class="sd"> The normalized schema mapping.</span>
</span><span id="L-363"><a href="#L-363"><span class="linenos">363</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-364"><a href="#L-364"><span class="linenos">364</span></a> <span class="n">normalized_mapping</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-365"><a href="#L-365"><span class="linenos">365</span></a> <span class="n">flattened_schema</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">depth</span><span class="o">=</span><span class="n">dict_depth</span><span class="p">(</span><span class="n">schema</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
</span><span id="L-366"><a href="#L-366"><span class="linenos">366</span></a>
</span><span id="L-367"><a href="#L-367"><span class="linenos">367</span></a> <span class="k">for</span> <span class="n">keys</span> <span class="ow">in</span> <span class="n">flattened_schema</span><span class="p">:</span>
</span><span id="L-368"><a href="#L-368"><span class="linenos">368</span></a> <span class="n">columns</span> <span class="o">=</span> <span class="n">nested_get</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="o">*</span><span class="nb">zip</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">keys</span><span class="p">))</span>
</span><span id="L-369"><a href="#L-369"><span class="linenos">369</span></a>
</span><span id="L-370"><a href="#L-370"><span class="linenos">370</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">columns</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="L-371"><a href="#L-371"><span class="linenos">371</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span>
</span><span id="L-372"><a href="#L-372"><span class="linenos">372</span></a> <span class="sa">f</span><span class="s2">&quot;Table </span><span class="si">{</span><span class="s1">&#39;.&#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">keys</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span><span class="si">}</span><span class="s2"> must match the schema&#39;s nesting level: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">flattened_schema</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="si">}</span><span class="s2">.&quot;</span>
</span><span id="L-373"><a href="#L-373"><span class="linenos">373</span></a> <span class="p">)</span>
</span><span id="L-374"><a href="#L-374"><span class="linenos">374</span></a>
</span><span id="L-375"><a href="#L-375"><span class="linenos">375</span></a> <span class="n">normalized_keys</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_normalize_name</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">is_table</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">]</span>
</span><span id="L-376"><a href="#L-376"><span class="linenos">376</span></a> <span class="k">for</span> <span class="n">column_name</span><span class="p">,</span> <span class="n">column_type</span> <span class="ow">in</span> <span class="n">columns</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="L-377"><a href="#L-377"><span class="linenos">377</span></a> <span class="n">nested_set</span><span class="p">(</span>
</span><span id="L-378"><a href="#L-378"><span class="linenos">378</span></a> <span class="n">normalized_mapping</span><span class="p">,</span>
</span><span id="L-379"><a href="#L-379"><span class="linenos">379</span></a> <span class="n">normalized_keys</span> <span class="o">+</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_normalize_name</span><span class="p">(</span><span class="n">column_name</span><span class="p">)],</span>
</span><span id="L-380"><a href="#L-380"><span class="linenos">380</span></a> <span class="n">column_type</span><span class="p">,</span>
</span><span id="L-381"><a href="#L-381"><span class="linenos">381</span></a> <span class="p">)</span>
</span><span id="L-382"><a href="#L-382"><span class="linenos">382</span></a>
</span><span id="L-383"><a href="#L-383"><span class="linenos">383</span></a> <span class="k">return</span> <span class="n">normalized_mapping</span>
</span><span id="L-384"><a href="#L-384"><span class="linenos">384</span></a>
</span><span id="L-385"><a href="#L-385"><span class="linenos">385</span></a> <span class="k">def</span> <span class="nf">_normalize_table</span><span class="p">(</span>
</span><span id="L-386"><a href="#L-386"><span class="linenos">386</span></a> <span class="bp">self</span><span class="p">,</span>
</span><span id="L-387"><a href="#L-387"><span class="linenos">387</span></a> <span class="n">table</span><span class="p">:</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span> <span class="o">|</span> <span class="nb">str</span><span class="p">,</span>
</span><span id="L-388"><a href="#L-388"><span class="linenos">388</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-389"><a href="#L-389"><span class="linenos">389</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-390"><a href="#L-390"><span class="linenos">390</span></a> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">:</span>
</span><span id="L-391"><a href="#L-391"><span class="linenos">391</span></a> <span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span>
</span><span id="L-392"><a href="#L-392"><span class="linenos">392</span></a> <span class="n">normalize</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">normalize</span> <span class="k">if</span> <span class="n">normalize</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">normalize</span>
</span><span id="L-393"><a href="#L-393"><span class="linenos">393</span></a>
</span><span id="L-394"><a href="#L-394"><span class="linenos">394</span></a> <span class="n">normalized_table</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">maybe_parse</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">into</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="n">normalize</span><span class="p">)</span>
</span><span id="L-395"><a href="#L-395"><span class="linenos">395</span></a>
</span><span id="L-396"><a href="#L-396"><span class="linenos">396</span></a> <span class="k">if</span> <span class="n">normalize</span><span class="p">:</span>
</span><span id="L-397"><a href="#L-397"><span class="linenos">397</span></a> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">exp</span><span class="o">.</span><span class="n">TABLE_PARTS</span><span class="p">:</span>
</span><span id="L-398"><a href="#L-398"><span class="linenos">398</span></a> <span class="n">value</span> <span class="o">=</span> <span class="n">normalized_table</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span>
</span><span id="L-399"><a href="#L-399"><span class="linenos">399</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">):</span>
</span><span id="L-400"><a href="#L-400"><span class="linenos">400</span></a> <span class="n">normalized_table</span><span class="o">.</span><span class="n">set</span><span class="p">(</span>
</span><span id="L-401"><a href="#L-401"><span class="linenos">401</span></a> <span class="n">arg</span><span class="p">,</span>
</span><span id="L-402"><a href="#L-402"><span class="linenos">402</span></a> <span class="n">normalize_name</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">is_table</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="n">normalize</span><span class="p">),</span>
</span><span id="L-403"><a href="#L-403"><span class="linenos">403</span></a> <span class="p">)</span>
</span><span id="L-404"><a href="#L-404"><span class="linenos">404</span></a>
</span><span id="L-405"><a href="#L-405"><span class="linenos">405</span></a> <span class="k">return</span> <span class="n">normalized_table</span>
</span><span id="L-406"><a href="#L-406"><span class="linenos">406</span></a>
</span><span id="L-407"><a href="#L-407"><span class="linenos">407</span></a> <span class="k">def</span> <span class="nf">_normalize_name</span><span class="p">(</span>
</span><span id="L-408"><a href="#L-408"><span class="linenos">408</span></a> <span class="bp">self</span><span class="p">,</span>
</span><span id="L-409"><a href="#L-409"><span class="linenos">409</span></a> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">,</span>
</span><span id="L-410"><a href="#L-410"><span class="linenos">410</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-411"><a href="#L-411"><span class="linenos">411</span></a> <span class="n">is_table</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-412"><a href="#L-412"><span class="linenos">412</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-413"><a href="#L-413"><span class="linenos">413</span></a> <span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span><span id="L-414"><a href="#L-414"><span class="linenos">414</span></a> <span class="k">return</span> <span class="n">normalize_name</span><span class="p">(</span>
</span><span id="L-415"><a href="#L-415"><span class="linenos">415</span></a> <span class="n">name</span><span class="p">,</span>
</span><span id="L-416"><a href="#L-416"><span class="linenos">416</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="L-417"><a href="#L-417"><span class="linenos">417</span></a> <span class="n">is_table</span><span class="o">=</span><span class="n">is_table</span><span class="p">,</span>
</span><span id="L-418"><a href="#L-418"><span class="linenos">418</span></a> <span class="n">normalize</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">normalize</span> <span class="k">if</span> <span class="n">normalize</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">normalize</span><span class="p">,</span>
</span><span id="L-419"><a href="#L-419"><span class="linenos">419</span></a> <span class="p">)</span><span class="o">.</span><span class="n">name</span>
</span><span id="L-420"><a href="#L-420"><span class="linenos">420</span></a>
</span><span id="L-421"><a href="#L-421"><span class="linenos">421</span></a> <span class="k">def</span> <span class="nf">depth</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span><span id="L-422"><a href="#L-422"><span class="linenos">422</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">empty</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span><span class="p">:</span>
</span><span id="L-423"><a href="#L-423"><span class="linenos">423</span></a> <span class="c1"># The columns themselves are a mapping, but we don&#39;t want to include those</span>
</span><span id="L-424"><a href="#L-424"><span class="linenos">424</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">depth</span><span class="p">()</span> <span class="o">-</span> <span class="mi">1</span>
</span><span id="L-425"><a href="#L-425"><span class="linenos">425</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span>
</span><span id="L-426"><a href="#L-426"><span class="linenos">426</span></a>
</span><span id="L-427"><a href="#L-427"><span class="linenos">427</span></a> <span class="k">def</span> <span class="nf">_to_data_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">DataType</span><span class="p">:</span>
</span><span id="L-428"><a href="#L-428"><span class="linenos">428</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-429"><a href="#L-429"><span class="linenos">429</span></a><span class="sd"> Convert a type represented as a string to the corresponding `sqlglot.exp.DataType` object.</span>
</span><span id="L-430"><a href="#L-430"><span class="linenos">430</span></a>
</span><span id="L-431"><a href="#L-431"><span class="linenos">431</span></a><span class="sd"> Args:</span>
</span><span id="L-432"><a href="#L-432"><span class="linenos">432</span></a><span class="sd"> schema_type: the type we want to convert.</span>
</span><span id="L-433"><a href="#L-433"><span class="linenos">433</span></a><span class="sd"> dialect: the SQL dialect that will be used to parse `schema_type`, if needed.</span>
</span><span id="L-434"><a href="#L-434"><span class="linenos">434</span></a>
</span><span id="L-435"><a href="#L-435"><span class="linenos">435</span></a><span class="sd"> Returns:</span>
</span><span id="L-436"><a href="#L-436"><span class="linenos">436</span></a><span class="sd"> The resulting expression type.</span>
</span><span id="L-437"><a href="#L-437"><span class="linenos">437</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-438"><a href="#L-438"><span class="linenos">438</span></a> <span class="k">if</span> <span class="n">schema_type</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">:</span>
</span><span id="L-439"><a href="#L-439"><span class="linenos">439</span></a> <span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span>
</span><span id="L-440"><a href="#L-440"><span class="linenos">440</span></a> <span class="n">udt</span> <span class="o">=</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">SUPPORTS_USER_DEFINED_TYPES</span>
</span><span id="L-441"><a href="#L-441"><span class="linenos">441</span></a>
</span><span id="L-442"><a href="#L-442"><span class="linenos">442</span></a> <span class="k">try</span><span class="p">:</span>
</span><span id="L-443"><a href="#L-443"><span class="linenos">443</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">DataType</span><span class="o">.</span><span class="n">build</span><span class="p">(</span><span class="n">schema_type</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">udt</span><span class="o">=</span><span class="n">udt</span><span class="p">)</span>
</span><span id="L-444"><a href="#L-444"><span class="linenos">444</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">[</span><span class="n">schema_type</span><span class="p">]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="L-445"><a href="#L-445"><span class="linenos">445</span></a> <span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
</span><span id="L-446"><a href="#L-446"><span class="linenos">446</span></a> <span class="n">in_dialect</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot; in dialect </span><span class="si">{</span><span class="n">dialect</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">if</span> <span class="n">dialect</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
</span><span id="L-447"><a href="#L-447"><span class="linenos">447</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to build type &#39;</span><span class="si">{</span><span class="n">schema_type</span><span class="si">}</span><span class="s2">&#39;</span><span class="si">{</span><span class="n">in_dialect</span><span class="si">}</span><span class="s2">.&quot;</span><span class="p">)</span>
</span><span id="L-448"><a href="#L-448"><span class="linenos">448</span></a>
</span><span id="L-449"><a href="#L-449"><span class="linenos">449</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">[</span><span class="n">schema_type</span><span class="p">]</span>
</span><span id="L-450"><a href="#L-450"><span class="linenos">450</span></a>
</span><span id="L-451"><a href="#L-451"><span class="linenos">451</span></a>
</span><span id="L-452"><a href="#L-452"><span class="linenos">452</span></a><span class="k">def</span> <span class="nf">normalize_name</span><span class="p">(</span>
</span><span id="L-453"><a href="#L-453"><span class="linenos">453</span></a> <span class="n">identifier</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">,</span>
</span><span id="L-454"><a href="#L-454"><span class="linenos">454</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-455"><a href="#L-455"><span class="linenos">455</span></a> <span class="n">is_table</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-456"><a href="#L-456"><span class="linenos">456</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-457"><a href="#L-457"><span class="linenos">457</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">:</span>
</span><span id="L-458"><a href="#L-458"><span class="linenos">458</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">identifier</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span><span id="L-459"><a href="#L-459"><span class="linenos">459</span></a> <span class="n">identifier</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">parse_identifier</span><span class="p">(</span><span class="n">identifier</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="L-460"><a href="#L-460"><span class="linenos">460</span></a>
</span><span id="L-461"><a href="#L-461"><span class="linenos">461</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">normalize</span><span class="p">:</span>
</span><span id="L-462"><a href="#L-462"><span class="linenos">462</span></a> <span class="k">return</span> <span class="n">identifier</span>
</span><span id="L-463"><a href="#L-463"><span class="linenos">463</span></a>
</span><span id="L-464"><a href="#L-464"><span class="linenos">464</span></a> <span class="c1"># this is used for normalize_identifier, bigquery has special rules pertaining tables</span>
</span><span id="L-465"><a href="#L-465"><span class="linenos">465</span></a> <span class="n">identifier</span><span class="o">.</span><span class="n">meta</span><span class="p">[</span><span class="s2">&quot;is_table&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">is_table</span>
</span><span id="L-466"><a href="#L-466"><span class="linenos">466</span></a> <span class="k">return</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">normalize_identifier</span><span class="p">(</span><span class="n">identifier</span><span class="p">)</span>
</span><span id="L-467"><a href="#L-467"><span class="linenos">467</span></a>
</span><span id="L-365"><a href="#L-365"><span class="linenos">365</span></a> <span class="n">flattened_schema</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">)</span>
</span><span id="L-366"><a href="#L-366"><span class="linenos">366</span></a> <span class="n">error_msg</span> <span class="o">=</span> <span class="s2">&quot;Table </span><span class="si">{}</span><span class="s2"> must match the schema&#39;s nesting level: </span><span class="si">{}</span><span class="s2">.&quot;</span>
</span><span id="L-367"><a href="#L-367"><span class="linenos">367</span></a>
</span><span id="L-368"><a href="#L-368"><span class="linenos">368</span></a> <span class="k">for</span> <span class="n">keys</span> <span class="ow">in</span> <span class="n">flattened_schema</span><span class="p">:</span>
</span><span id="L-369"><a href="#L-369"><span class="linenos">369</span></a> <span class="n">columns</span> <span class="o">=</span> <span class="n">nested_get</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="o">*</span><span class="nb">zip</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">keys</span><span class="p">))</span>
</span><span id="L-370"><a href="#L-370"><span class="linenos">370</span></a>
</span><span id="L-371"><a href="#L-371"><span class="linenos">371</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">columns</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="L-372"><a href="#L-372"><span class="linenos">372</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span><span class="n">error_msg</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">keys</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]),</span> <span class="nb">len</span><span class="p">(</span><span class="n">flattened_schema</span><span class="p">[</span><span class="mi">0</span><span class="p">])))</span>
</span><span id="L-373"><a href="#L-373"><span class="linenos">373</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">first</span><span class="p">(</span><span class="n">columns</span><span class="o">.</span><span class="n">values</span><span class="p">()),</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="L-374"><a href="#L-374"><span class="linenos">374</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span>
</span><span id="L-375"><a href="#L-375"><span class="linenos">375</span></a> <span class="n">error_msg</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
</span><span id="L-376"><a href="#L-376"><span class="linenos">376</span></a> <span class="s2">&quot;.&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">keys</span> <span class="o">+</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">columns</span><span class="p">)[</span><span class="mi">0</span><span class="p">]),</span> <span class="nb">len</span><span class="p">(</span><span class="n">flattened_schema</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</span><span id="L-377"><a href="#L-377"><span class="linenos">377</span></a> <span class="p">),</span>
</span><span id="L-378"><a href="#L-378"><span class="linenos">378</span></a> <span class="p">)</span>
</span><span id="L-379"><a href="#L-379"><span class="linenos">379</span></a>
</span><span id="L-380"><a href="#L-380"><span class="linenos">380</span></a> <span class="n">normalized_keys</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_normalize_name</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">is_table</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">]</span>
</span><span id="L-381"><a href="#L-381"><span class="linenos">381</span></a> <span class="k">for</span> <span class="n">column_name</span><span class="p">,</span> <span class="n">column_type</span> <span class="ow">in</span> <span class="n">columns</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="L-382"><a href="#L-382"><span class="linenos">382</span></a> <span class="n">nested_set</span><span class="p">(</span>
</span><span id="L-383"><a href="#L-383"><span class="linenos">383</span></a> <span class="n">normalized_mapping</span><span class="p">,</span>
</span><span id="L-384"><a href="#L-384"><span class="linenos">384</span></a> <span class="n">normalized_keys</span> <span class="o">+</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_normalize_name</span><span class="p">(</span><span class="n">column_name</span><span class="p">)],</span>
</span><span id="L-385"><a href="#L-385"><span class="linenos">385</span></a> <span class="n">column_type</span><span class="p">,</span>
</span><span id="L-386"><a href="#L-386"><span class="linenos">386</span></a> <span class="p">)</span>
</span><span id="L-387"><a href="#L-387"><span class="linenos">387</span></a>
</span><span id="L-388"><a href="#L-388"><span class="linenos">388</span></a> <span class="k">return</span> <span class="n">normalized_mapping</span>
</span><span id="L-389"><a href="#L-389"><span class="linenos">389</span></a>
</span><span id="L-390"><a href="#L-390"><span class="linenos">390</span></a> <span class="k">def</span> <span class="nf">_normalize_table</span><span class="p">(</span>
</span><span id="L-391"><a href="#L-391"><span class="linenos">391</span></a> <span class="bp">self</span><span class="p">,</span>
</span><span id="L-392"><a href="#L-392"><span class="linenos">392</span></a> <span class="n">table</span><span class="p">:</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span> <span class="o">|</span> <span class="nb">str</span><span class="p">,</span>
</span><span id="L-393"><a href="#L-393"><span class="linenos">393</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-394"><a href="#L-394"><span class="linenos">394</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-395"><a href="#L-395"><span class="linenos">395</span></a> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">:</span>
</span><span id="L-396"><a href="#L-396"><span class="linenos">396</span></a> <span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span>
</span><span id="L-397"><a href="#L-397"><span class="linenos">397</span></a> <span class="n">normalize</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">normalize</span> <span class="k">if</span> <span class="n">normalize</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">normalize</span>
</span><span id="L-398"><a href="#L-398"><span class="linenos">398</span></a>
</span><span id="L-399"><a href="#L-399"><span class="linenos">399</span></a> <span class="n">normalized_table</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">maybe_parse</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">into</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="n">normalize</span><span class="p">)</span>
</span><span id="L-400"><a href="#L-400"><span class="linenos">400</span></a>
</span><span id="L-401"><a href="#L-401"><span class="linenos">401</span></a> <span class="k">if</span> <span class="n">normalize</span><span class="p">:</span>
</span><span id="L-402"><a href="#L-402"><span class="linenos">402</span></a> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">exp</span><span class="o">.</span><span class="n">TABLE_PARTS</span><span class="p">:</span>
</span><span id="L-403"><a href="#L-403"><span class="linenos">403</span></a> <span class="n">value</span> <span class="o">=</span> <span class="n">normalized_table</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span>
</span><span id="L-404"><a href="#L-404"><span class="linenos">404</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">):</span>
</span><span id="L-405"><a href="#L-405"><span class="linenos">405</span></a> <span class="n">normalized_table</span><span class="o">.</span><span class="n">set</span><span class="p">(</span>
</span><span id="L-406"><a href="#L-406"><span class="linenos">406</span></a> <span class="n">arg</span><span class="p">,</span>
</span><span id="L-407"><a href="#L-407"><span class="linenos">407</span></a> <span class="n">normalize_name</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">is_table</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="n">normalize</span><span class="p">),</span>
</span><span id="L-408"><a href="#L-408"><span class="linenos">408</span></a> <span class="p">)</span>
</span><span id="L-409"><a href="#L-409"><span class="linenos">409</span></a>
</span><span id="L-410"><a href="#L-410"><span class="linenos">410</span></a> <span class="k">return</span> <span class="n">normalized_table</span>
</span><span id="L-411"><a href="#L-411"><span class="linenos">411</span></a>
</span><span id="L-412"><a href="#L-412"><span class="linenos">412</span></a> <span class="k">def</span> <span class="nf">_normalize_name</span><span class="p">(</span>
</span><span id="L-413"><a href="#L-413"><span class="linenos">413</span></a> <span class="bp">self</span><span class="p">,</span>
</span><span id="L-414"><a href="#L-414"><span class="linenos">414</span></a> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">,</span>
</span><span id="L-415"><a href="#L-415"><span class="linenos">415</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-416"><a href="#L-416"><span class="linenos">416</span></a> <span class="n">is_table</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-417"><a href="#L-417"><span class="linenos">417</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-418"><a href="#L-418"><span class="linenos">418</span></a> <span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span><span id="L-419"><a href="#L-419"><span class="linenos">419</span></a> <span class="k">return</span> <span class="n">normalize_name</span><span class="p">(</span>
</span><span id="L-420"><a href="#L-420"><span class="linenos">420</span></a> <span class="n">name</span><span class="p">,</span>
</span><span id="L-421"><a href="#L-421"><span class="linenos">421</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="L-422"><a href="#L-422"><span class="linenos">422</span></a> <span class="n">is_table</span><span class="o">=</span><span class="n">is_table</span><span class="p">,</span>
</span><span id="L-423"><a href="#L-423"><span class="linenos">423</span></a> <span class="n">normalize</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">normalize</span> <span class="k">if</span> <span class="n">normalize</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">normalize</span><span class="p">,</span>
</span><span id="L-424"><a href="#L-424"><span class="linenos">424</span></a> <span class="p">)</span><span class="o">.</span><span class="n">name</span>
</span><span id="L-425"><a href="#L-425"><span class="linenos">425</span></a>
</span><span id="L-426"><a href="#L-426"><span class="linenos">426</span></a> <span class="k">def</span> <span class="nf">depth</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span><span id="L-427"><a href="#L-427"><span class="linenos">427</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">empty</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span><span class="p">:</span>
</span><span id="L-428"><a href="#L-428"><span class="linenos">428</span></a> <span class="c1"># The columns themselves are a mapping, but we don&#39;t want to include those</span>
</span><span id="L-429"><a href="#L-429"><span class="linenos">429</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">depth</span><span class="p">()</span> <span class="o">-</span> <span class="mi">1</span>
</span><span id="L-430"><a href="#L-430"><span class="linenos">430</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span>
</span><span id="L-431"><a href="#L-431"><span class="linenos">431</span></a>
</span><span id="L-432"><a href="#L-432"><span class="linenos">432</span></a> <span class="k">def</span> <span class="nf">_to_data_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">DataType</span><span class="p">:</span>
</span><span id="L-433"><a href="#L-433"><span class="linenos">433</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-434"><a href="#L-434"><span class="linenos">434</span></a><span class="sd"> Convert a type represented as a string to the corresponding `sqlglot.exp.DataType` object.</span>
</span><span id="L-435"><a href="#L-435"><span class="linenos">435</span></a>
</span><span id="L-436"><a href="#L-436"><span class="linenos">436</span></a><span class="sd"> Args:</span>
</span><span id="L-437"><a href="#L-437"><span class="linenos">437</span></a><span class="sd"> schema_type: the type we want to convert.</span>
</span><span id="L-438"><a href="#L-438"><span class="linenos">438</span></a><span class="sd"> dialect: the SQL dialect that will be used to parse `schema_type`, if needed.</span>
</span><span id="L-439"><a href="#L-439"><span class="linenos">439</span></a>
</span><span id="L-440"><a href="#L-440"><span class="linenos">440</span></a><span class="sd"> Returns:</span>
</span><span id="L-441"><a href="#L-441"><span class="linenos">441</span></a><span class="sd"> The resulting expression type.</span>
</span><span id="L-442"><a href="#L-442"><span class="linenos">442</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-443"><a href="#L-443"><span class="linenos">443</span></a> <span class="k">if</span> <span class="n">schema_type</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">:</span>
</span><span id="L-444"><a href="#L-444"><span class="linenos">444</span></a> <span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span>
</span><span id="L-445"><a href="#L-445"><span class="linenos">445</span></a> <span class="n">udt</span> <span class="o">=</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">SUPPORTS_USER_DEFINED_TYPES</span>
</span><span id="L-446"><a href="#L-446"><span class="linenos">446</span></a>
</span><span id="L-447"><a href="#L-447"><span class="linenos">447</span></a> <span class="k">try</span><span class="p">:</span>
</span><span id="L-448"><a href="#L-448"><span class="linenos">448</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">DataType</span><span class="o">.</span><span class="n">build</span><span class="p">(</span><span class="n">schema_type</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">udt</span><span class="o">=</span><span class="n">udt</span><span class="p">)</span>
</span><span id="L-449"><a href="#L-449"><span class="linenos">449</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">[</span><span class="n">schema_type</span><span class="p">]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="L-450"><a href="#L-450"><span class="linenos">450</span></a> <span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
</span><span id="L-451"><a href="#L-451"><span class="linenos">451</span></a> <span class="n">in_dialect</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot; in dialect </span><span class="si">{</span><span class="n">dialect</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">if</span> <span class="n">dialect</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
</span><span id="L-452"><a href="#L-452"><span class="linenos">452</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to build type &#39;</span><span class="si">{</span><span class="n">schema_type</span><span class="si">}</span><span class="s2">&#39;</span><span class="si">{</span><span class="n">in_dialect</span><span class="si">}</span><span class="s2">.&quot;</span><span class="p">)</span>
</span><span id="L-453"><a href="#L-453"><span class="linenos">453</span></a>
</span><span id="L-454"><a href="#L-454"><span class="linenos">454</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">[</span><span class="n">schema_type</span><span class="p">]</span>
</span><span id="L-455"><a href="#L-455"><span class="linenos">455</span></a>
</span><span id="L-456"><a href="#L-456"><span class="linenos">456</span></a>
</span><span id="L-457"><a href="#L-457"><span class="linenos">457</span></a><span class="k">def</span> <span class="nf">normalize_name</span><span class="p">(</span>
</span><span id="L-458"><a href="#L-458"><span class="linenos">458</span></a> <span class="n">identifier</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">,</span>
</span><span id="L-459"><a href="#L-459"><span class="linenos">459</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-460"><a href="#L-460"><span class="linenos">460</span></a> <span class="n">is_table</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-461"><a href="#L-461"><span class="linenos">461</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-462"><a href="#L-462"><span class="linenos">462</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">:</span>
</span><span id="L-463"><a href="#L-463"><span class="linenos">463</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">identifier</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span><span id="L-464"><a href="#L-464"><span class="linenos">464</span></a> <span class="n">identifier</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">parse_identifier</span><span class="p">(</span><span class="n">identifier</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="L-465"><a href="#L-465"><span class="linenos">465</span></a>
</span><span id="L-466"><a href="#L-466"><span class="linenos">466</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">normalize</span><span class="p">:</span>
</span><span id="L-467"><a href="#L-467"><span class="linenos">467</span></a> <span class="k">return</span> <span class="n">identifier</span>
</span><span id="L-468"><a href="#L-468"><span class="linenos">468</span></a>
</span><span id="L-469"><a href="#L-469"><span class="linenos">469</span></a><span class="k">def</span> <span class="nf">ensure_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">:</span> <span class="n">Schema</span> <span class="o">|</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">],</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Schema</span><span class="p">:</span>
</span><span id="L-470"><a href="#L-470"><span class="linenos">470</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">Schema</span><span class="p">):</span>
</span><span id="L-471"><a href="#L-471"><span class="linenos">471</span></a> <span class="k">return</span> <span class="n">schema</span>
</span><span id="L-469"><a href="#L-469"><span class="linenos">469</span></a> <span class="c1"># this is used for normalize_identifier, bigquery has special rules pertaining tables</span>
</span><span id="L-470"><a href="#L-470"><span class="linenos">470</span></a> <span class="n">identifier</span><span class="o">.</span><span class="n">meta</span><span class="p">[</span><span class="s2">&quot;is_table&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">is_table</span>
</span><span id="L-471"><a href="#L-471"><span class="linenos">471</span></a> <span class="k">return</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">normalize_identifier</span><span class="p">(</span><span class="n">identifier</span><span class="p">)</span>
</span><span id="L-472"><a href="#L-472"><span class="linenos">472</span></a>
</span><span id="L-473"><a href="#L-473"><span class="linenos">473</span></a> <span class="k">return</span> <span class="n">MappingSchema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span><span id="L-474"><a href="#L-474"><span class="linenos">474</span></a>
</span><span id="L-475"><a href="#L-475"><span class="linenos">475</span></a>
</span><span id="L-476"><a href="#L-476"><span class="linenos">476</span></a><span class="k">def</span> <span class="nf">ensure_column_mapping</span><span class="p">(</span><span class="n">mapping</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">ColumnMapping</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">:</span>
</span><span id="L-477"><a href="#L-477"><span class="linenos">477</span></a> <span class="k">if</span> <span class="n">mapping</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span><span id="L-478"><a href="#L-478"><span class="linenos">478</span></a> <span class="k">return</span> <span class="p">{}</span>
</span><span id="L-479"><a href="#L-479"><span class="linenos">479</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="L-480"><a href="#L-480"><span class="linenos">480</span></a> <span class="k">return</span> <span class="n">mapping</span>
</span><span id="L-481"><a href="#L-481"><span class="linenos">481</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span><span id="L-482"><a href="#L-482"><span class="linenos">482</span></a> <span class="n">col_name_type_strs</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">mapping</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
</span><span id="L-483"><a href="#L-483"><span class="linenos">483</span></a> <span class="k">return</span> <span class="p">{</span>
</span><span id="L-484"><a href="#L-484"><span class="linenos">484</span></a> <span class="n">name_type_str</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span> <span class="n">name_type_str</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span><span id="L-485"><a href="#L-485"><span class="linenos">485</span></a> <span class="k">for</span> <span class="n">name_type_str</span> <span class="ow">in</span> <span class="n">col_name_type_strs</span>
</span><span id="L-486"><a href="#L-486"><span class="linenos">486</span></a> <span class="p">}</span>
</span><span id="L-487"><a href="#L-487"><span class="linenos">487</span></a> <span class="c1"># Check if mapping looks like a DataFrame StructType</span>
</span><span id="L-488"><a href="#L-488"><span class="linenos">488</span></a> <span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="s2">&quot;simpleString&quot;</span><span class="p">):</span>
</span><span id="L-489"><a href="#L-489"><span class="linenos">489</span></a> <span class="k">return</span> <span class="p">{</span><span class="n">struct_field</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">struct_field</span><span class="o">.</span><span class="n">dataType</span><span class="o">.</span><span class="n">simpleString</span><span class="p">()</span> <span class="k">for</span> <span class="n">struct_field</span> <span class="ow">in</span> <span class="n">mapping</span><span class="p">}</span>
</span><span id="L-490"><a href="#L-490"><span class="linenos">490</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
</span><span id="L-491"><a href="#L-491"><span class="linenos">491</span></a> <span class="k">return</span> <span class="p">{</span><span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span> <span class="kc">None</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">mapping</span><span class="p">}</span>
</span><span id="L-492"><a href="#L-492"><span class="linenos">492</span></a>
</span><span id="L-493"><a href="#L-493"><span class="linenos">493</span></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid mapping provided: </span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="n">mapping</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="L-494"><a href="#L-494"><span class="linenos">494</span></a>
</span><span id="L-495"><a href="#L-495"><span class="linenos">495</span></a>
</span><span id="L-496"><a href="#L-496"><span class="linenos">496</span></a><span class="k">def</span> <span class="nf">flatten_schema</span><span class="p">(</span>
</span><span id="L-497"><a href="#L-497"><span class="linenos">497</span></a> <span class="n">schema</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">depth</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">keys</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-498"><a href="#L-498"><span class="linenos">498</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]:</span>
</span><span id="L-499"><a href="#L-499"><span class="linenos">499</span></a> <span class="n">tables</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-500"><a href="#L-500"><span class="linenos">500</span></a> <span class="n">keys</span> <span class="o">=</span> <span class="n">keys</span> <span class="ow">or</span> <span class="p">[]</span>
</span><span id="L-501"><a href="#L-501"><span class="linenos">501</span></a>
</span><span id="L-502"><a href="#L-502"><span class="linenos">502</span></a> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">schema</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="L-503"><a href="#L-503"><span class="linenos">503</span></a> <span class="k">if</span> <span class="n">depth</span> <span class="o">&gt;=</span> <span class="mi">2</span><span class="p">:</span>
</span><span id="L-504"><a href="#L-504"><span class="linenos">504</span></a> <span class="n">tables</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">flatten_schema</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">depth</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="n">keys</span> <span class="o">+</span> <span class="p">[</span><span class="n">k</span><span class="p">]))</span>
</span><span id="L-505"><a href="#L-505"><span class="linenos">505</span></a> <span class="k">elif</span> <span class="n">depth</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="L-506"><a href="#L-506"><span class="linenos">506</span></a> <span class="n">tables</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">keys</span> <span class="o">+</span> <span class="p">[</span><span class="n">k</span><span class="p">])</span>
</span><span id="L-473"><a href="#L-473"><span class="linenos">473</span></a>
</span><span id="L-474"><a href="#L-474"><span class="linenos">474</span></a><span class="k">def</span> <span class="nf">ensure_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">:</span> <span class="n">Schema</span> <span class="o">|</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">],</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Schema</span><span class="p">:</span>
</span><span id="L-475"><a href="#L-475"><span class="linenos">475</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">Schema</span><span class="p">):</span>
</span><span id="L-476"><a href="#L-476"><span class="linenos">476</span></a> <span class="k">return</span> <span class="n">schema</span>
</span><span id="L-477"><a href="#L-477"><span class="linenos">477</span></a>
</span><span id="L-478"><a href="#L-478"><span class="linenos">478</span></a> <span class="k">return</span> <span class="n">MappingSchema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span><span id="L-479"><a href="#L-479"><span class="linenos">479</span></a>
</span><span id="L-480"><a href="#L-480"><span class="linenos">480</span></a>
</span><span id="L-481"><a href="#L-481"><span class="linenos">481</span></a><span class="k">def</span> <span class="nf">ensure_column_mapping</span><span class="p">(</span><span class="n">mapping</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">ColumnMapping</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">:</span>
</span><span id="L-482"><a href="#L-482"><span class="linenos">482</span></a> <span class="k">if</span> <span class="n">mapping</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span><span id="L-483"><a href="#L-483"><span class="linenos">483</span></a> <span class="k">return</span> <span class="p">{}</span>
</span><span id="L-484"><a href="#L-484"><span class="linenos">484</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="L-485"><a href="#L-485"><span class="linenos">485</span></a> <span class="k">return</span> <span class="n">mapping</span>
</span><span id="L-486"><a href="#L-486"><span class="linenos">486</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span><span id="L-487"><a href="#L-487"><span class="linenos">487</span></a> <span class="n">col_name_type_strs</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">mapping</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
</span><span id="L-488"><a href="#L-488"><span class="linenos">488</span></a> <span class="k">return</span> <span class="p">{</span>
</span><span id="L-489"><a href="#L-489"><span class="linenos">489</span></a> <span class="n">name_type_str</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span> <span class="n">name_type_str</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span><span id="L-490"><a href="#L-490"><span class="linenos">490</span></a> <span class="k">for</span> <span class="n">name_type_str</span> <span class="ow">in</span> <span class="n">col_name_type_strs</span>
</span><span id="L-491"><a href="#L-491"><span class="linenos">491</span></a> <span class="p">}</span>
</span><span id="L-492"><a href="#L-492"><span class="linenos">492</span></a> <span class="c1"># Check if mapping looks like a DataFrame StructType</span>
</span><span id="L-493"><a href="#L-493"><span class="linenos">493</span></a> <span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="s2">&quot;simpleString&quot;</span><span class="p">):</span>
</span><span id="L-494"><a href="#L-494"><span class="linenos">494</span></a> <span class="k">return</span> <span class="p">{</span><span class="n">struct_field</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">struct_field</span><span class="o">.</span><span class="n">dataType</span><span class="o">.</span><span class="n">simpleString</span><span class="p">()</span> <span class="k">for</span> <span class="n">struct_field</span> <span class="ow">in</span> <span class="n">mapping</span><span class="p">}</span>
</span><span id="L-495"><a href="#L-495"><span class="linenos">495</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
</span><span id="L-496"><a href="#L-496"><span class="linenos">496</span></a> <span class="k">return</span> <span class="p">{</span><span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span> <span class="kc">None</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">mapping</span><span class="p">}</span>
</span><span id="L-497"><a href="#L-497"><span class="linenos">497</span></a>
</span><span id="L-498"><a href="#L-498"><span class="linenos">498</span></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid mapping provided: </span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="n">mapping</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="L-499"><a href="#L-499"><span class="linenos">499</span></a>
</span><span id="L-500"><a href="#L-500"><span class="linenos">500</span></a>
</span><span id="L-501"><a href="#L-501"><span class="linenos">501</span></a><span class="k">def</span> <span class="nf">flatten_schema</span><span class="p">(</span>
</span><span id="L-502"><a href="#L-502"><span class="linenos">502</span></a> <span class="n">schema</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">depth</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">keys</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-503"><a href="#L-503"><span class="linenos">503</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]:</span>
</span><span id="L-504"><a href="#L-504"><span class="linenos">504</span></a> <span class="n">tables</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-505"><a href="#L-505"><span class="linenos">505</span></a> <span class="n">keys</span> <span class="o">=</span> <span class="n">keys</span> <span class="ow">or</span> <span class="p">[]</span>
</span><span id="L-506"><a href="#L-506"><span class="linenos">506</span></a> <span class="n">depth</span> <span class="o">=</span> <span class="n">dict_depth</span><span class="p">(</span><span class="n">schema</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">depth</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">depth</span>
</span><span id="L-507"><a href="#L-507"><span class="linenos">507</span></a>
</span><span id="L-508"><a href="#L-508"><span class="linenos">508</span></a> <span class="k">return</span> <span class="n">tables</span>
</span><span id="L-509"><a href="#L-509"><span class="linenos">509</span></a>
</span><span id="L-510"><a href="#L-510"><span class="linenos">510</span></a>
</span><span id="L-511"><a href="#L-511"><span class="linenos">511</span></a><span class="k">def</span> <span class="nf">nested_get</span><span class="p">(</span>
</span><span id="L-512"><a href="#L-512"><span class="linenos">512</span></a> <span class="n">d</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="o">*</span><span class="n">path</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span> <span class="n">raise_on_missing</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>
</span><span id="L-513"><a href="#L-513"><span class="linenos">513</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">]:</span>
</span><span id="L-514"><a href="#L-514"><span class="linenos">514</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-515"><a href="#L-515"><span class="linenos">515</span></a><span class="sd"> Get a value for a nested dictionary.</span>
</span><span id="L-508"><a href="#L-508"><span class="linenos">508</span></a> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">schema</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="L-509"><a href="#L-509"><span class="linenos">509</span></a> <span class="k">if</span> <span class="n">depth</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="L-510"><a href="#L-510"><span class="linenos">510</span></a> <span class="n">tables</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">keys</span> <span class="o">+</span> <span class="p">[</span><span class="n">k</span><span class="p">])</span>
</span><span id="L-511"><a href="#L-511"><span class="linenos">511</span></a> <span class="k">elif</span> <span class="n">depth</span> <span class="o">&gt;=</span> <span class="mi">2</span><span class="p">:</span>
</span><span id="L-512"><a href="#L-512"><span class="linenos">512</span></a> <span class="n">tables</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">flatten_schema</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">depth</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="n">keys</span> <span class="o">+</span> <span class="p">[</span><span class="n">k</span><span class="p">]))</span>
</span><span id="L-513"><a href="#L-513"><span class="linenos">513</span></a>
</span><span id="L-514"><a href="#L-514"><span class="linenos">514</span></a> <span class="k">return</span> <span class="n">tables</span>
</span><span id="L-515"><a href="#L-515"><span class="linenos">515</span></a>
</span><span id="L-516"><a href="#L-516"><span class="linenos">516</span></a>
</span><span id="L-517"><a href="#L-517"><span class="linenos">517</span></a><span class="sd"> Args:</span>
</span><span id="L-518"><a href="#L-518"><span class="linenos">518</span></a><span class="sd"> d: the dictionary to search.</span>
</span><span id="L-519"><a href="#L-519"><span class="linenos">519</span></a><span class="sd"> *path: tuples of (name, key), where:</span>
</span><span id="L-520"><a href="#L-520"><span class="linenos">520</span></a><span class="sd"> `key` is the key in the dictionary to get.</span>
</span><span id="L-521"><a href="#L-521"><span class="linenos">521</span></a><span class="sd"> `name` is a string to use in the error if `key` isn&#39;t found.</span>
</span><span id="L-517"><a href="#L-517"><span class="linenos">517</span></a><span class="k">def</span> <span class="nf">nested_get</span><span class="p">(</span>
</span><span id="L-518"><a href="#L-518"><span class="linenos">518</span></a> <span class="n">d</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="o">*</span><span class="n">path</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span> <span class="n">raise_on_missing</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>
</span><span id="L-519"><a href="#L-519"><span class="linenos">519</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">]:</span>
</span><span id="L-520"><a href="#L-520"><span class="linenos">520</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-521"><a href="#L-521"><span class="linenos">521</span></a><span class="sd"> Get a value for a nested dictionary.</span>
</span><span id="L-522"><a href="#L-522"><span class="linenos">522</span></a>
</span><span id="L-523"><a href="#L-523"><span class="linenos">523</span></a><span class="sd"> Returns:</span>
</span><span id="L-524"><a href="#L-524"><span class="linenos">524</span></a><span class="sd"> The value or None if it doesn&#39;t exist.</span>
</span><span id="L-525"><a href="#L-525"><span class="linenos">525</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-526"><a href="#L-526"><span class="linenos">526</span></a> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">path</span><span class="p">:</span>
</span><span id="L-527"><a href="#L-527"><span class="linenos">527</span></a> <span class="n">d</span> <span class="o">=</span> <span class="n">d</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="c1"># type: ignore</span>
</span><span id="L-528"><a href="#L-528"><span class="linenos">528</span></a> <span class="k">if</span> <span class="n">d</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span><span id="L-529"><a href="#L-529"><span class="linenos">529</span></a> <span class="k">if</span> <span class="n">raise_on_missing</span><span class="p">:</span>
</span><span id="L-530"><a href="#L-530"><span class="linenos">530</span></a> <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;table&quot;</span> <span class="k">if</span> <span class="n">name</span> <span class="o">==</span> <span class="s2">&quot;this&quot;</span> <span class="k">else</span> <span class="n">name</span>
</span><span id="L-531"><a href="#L-531"><span class="linenos">531</span></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unknown </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="L-532"><a href="#L-532"><span class="linenos">532</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="L-533"><a href="#L-533"><span class="linenos">533</span></a>
</span><span id="L-534"><a href="#L-534"><span class="linenos">534</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="L-535"><a href="#L-535"><span class="linenos">535</span></a>
</span><span id="L-536"><a href="#L-536"><span class="linenos">536</span></a>
</span><span id="L-537"><a href="#L-537"><span class="linenos">537</span></a><span class="k">def</span> <span class="nf">nested_set</span><span class="p">(</span><span class="n">d</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">keys</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Sequence</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">value</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">:</span>
</span><span id="L-538"><a href="#L-538"><span class="linenos">538</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-539"><a href="#L-539"><span class="linenos">539</span></a><span class="sd"> In-place set a value for a nested dictionary</span>
</span><span id="L-540"><a href="#L-540"><span class="linenos">540</span></a>
</span><span id="L-541"><a href="#L-541"><span class="linenos">541</span></a><span class="sd"> Example:</span>
</span><span id="L-542"><a href="#L-542"><span class="linenos">542</span></a><span class="sd"> &gt;&gt;&gt; nested_set({}, [&quot;top_key&quot;, &quot;second_key&quot;], &quot;value&quot;)</span>
</span><span id="L-543"><a href="#L-543"><span class="linenos">543</span></a><span class="sd"> {&#39;top_key&#39;: {&#39;second_key&#39;: &#39;value&#39;}}</span>
</span><span id="L-544"><a href="#L-544"><span class="linenos">544</span></a>
</span><span id="L-545"><a href="#L-545"><span class="linenos">545</span></a><span class="sd"> &gt;&gt;&gt; nested_set({&quot;top_key&quot;: {&quot;third_key&quot;: &quot;third_value&quot;}}, [&quot;top_key&quot;, &quot;second_key&quot;], &quot;value&quot;)</span>
</span><span id="L-546"><a href="#L-546"><span class="linenos">546</span></a><span class="sd"> {&#39;top_key&#39;: {&#39;third_key&#39;: &#39;third_value&#39;, &#39;second_key&#39;: &#39;value&#39;}}</span>
</span><span id="L-547"><a href="#L-547"><span class="linenos">547</span></a>
</span><span id="L-548"><a href="#L-548"><span class="linenos">548</span></a><span class="sd"> Args:</span>
</span><span id="L-549"><a href="#L-549"><span class="linenos">549</span></a><span class="sd"> d: dictionary to update.</span>
</span><span id="L-550"><a href="#L-550"><span class="linenos">550</span></a><span class="sd"> keys: the keys that makeup the path to `value`.</span>
</span><span id="L-551"><a href="#L-551"><span class="linenos">551</span></a><span class="sd"> value: the value to set in the dictionary for the given key path.</span>
</span><span id="L-552"><a href="#L-552"><span class="linenos">552</span></a>
</span><span id="L-553"><a href="#L-553"><span class="linenos">553</span></a><span class="sd"> Returns:</span>
</span><span id="L-554"><a href="#L-554"><span class="linenos">554</span></a><span class="sd"> The (possibly) updated dictionary.</span>
</span><span id="L-555"><a href="#L-555"><span class="linenos">555</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-556"><a href="#L-556"><span class="linenos">556</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="L-557"><a href="#L-557"><span class="linenos">557</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="L-523"><a href="#L-523"><span class="linenos">523</span></a><span class="sd"> Args:</span>
</span><span id="L-524"><a href="#L-524"><span class="linenos">524</span></a><span class="sd"> d: the dictionary to search.</span>
</span><span id="L-525"><a href="#L-525"><span class="linenos">525</span></a><span class="sd"> *path: tuples of (name, key), where:</span>
</span><span id="L-526"><a href="#L-526"><span class="linenos">526</span></a><span class="sd"> `key` is the key in the dictionary to get.</span>
</span><span id="L-527"><a href="#L-527"><span class="linenos">527</span></a><span class="sd"> `name` is a string to use in the error if `key` isn&#39;t found.</span>
</span><span id="L-528"><a href="#L-528"><span class="linenos">528</span></a>
</span><span id="L-529"><a href="#L-529"><span class="linenos">529</span></a><span class="sd"> Returns:</span>
</span><span id="L-530"><a href="#L-530"><span class="linenos">530</span></a><span class="sd"> The value or None if it doesn&#39;t exist.</span>
</span><span id="L-531"><a href="#L-531"><span class="linenos">531</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-532"><a href="#L-532"><span class="linenos">532</span></a> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">path</span><span class="p">:</span>
</span><span id="L-533"><a href="#L-533"><span class="linenos">533</span></a> <span class="n">d</span> <span class="o">=</span> <span class="n">d</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="c1"># type: ignore</span>
</span><span id="L-534"><a href="#L-534"><span class="linenos">534</span></a> <span class="k">if</span> <span class="n">d</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span><span id="L-535"><a href="#L-535"><span class="linenos">535</span></a> <span class="k">if</span> <span class="n">raise_on_missing</span><span class="p">:</span>
</span><span id="L-536"><a href="#L-536"><span class="linenos">536</span></a> <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;table&quot;</span> <span class="k">if</span> <span class="n">name</span> <span class="o">==</span> <span class="s2">&quot;this&quot;</span> <span class="k">else</span> <span class="n">name</span>
</span><span id="L-537"><a href="#L-537"><span class="linenos">537</span></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unknown </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="L-538"><a href="#L-538"><span class="linenos">538</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="L-539"><a href="#L-539"><span class="linenos">539</span></a>
</span><span id="L-540"><a href="#L-540"><span class="linenos">540</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="L-541"><a href="#L-541"><span class="linenos">541</span></a>
</span><span id="L-542"><a href="#L-542"><span class="linenos">542</span></a>
</span><span id="L-543"><a href="#L-543"><span class="linenos">543</span></a><span class="k">def</span> <span class="nf">nested_set</span><span class="p">(</span><span class="n">d</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">keys</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Sequence</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">value</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">:</span>
</span><span id="L-544"><a href="#L-544"><span class="linenos">544</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-545"><a href="#L-545"><span class="linenos">545</span></a><span class="sd"> In-place set a value for a nested dictionary</span>
</span><span id="L-546"><a href="#L-546"><span class="linenos">546</span></a>
</span><span id="L-547"><a href="#L-547"><span class="linenos">547</span></a><span class="sd"> Example:</span>
</span><span id="L-548"><a href="#L-548"><span class="linenos">548</span></a><span class="sd"> &gt;&gt;&gt; nested_set({}, [&quot;top_key&quot;, &quot;second_key&quot;], &quot;value&quot;)</span>
</span><span id="L-549"><a href="#L-549"><span class="linenos">549</span></a><span class="sd"> {&#39;top_key&#39;: {&#39;second_key&#39;: &#39;value&#39;}}</span>
</span><span id="L-550"><a href="#L-550"><span class="linenos">550</span></a>
</span><span id="L-551"><a href="#L-551"><span class="linenos">551</span></a><span class="sd"> &gt;&gt;&gt; nested_set({&quot;top_key&quot;: {&quot;third_key&quot;: &quot;third_value&quot;}}, [&quot;top_key&quot;, &quot;second_key&quot;], &quot;value&quot;)</span>
</span><span id="L-552"><a href="#L-552"><span class="linenos">552</span></a><span class="sd"> {&#39;top_key&#39;: {&#39;third_key&#39;: &#39;third_value&#39;, &#39;second_key&#39;: &#39;value&#39;}}</span>
</span><span id="L-553"><a href="#L-553"><span class="linenos">553</span></a>
</span><span id="L-554"><a href="#L-554"><span class="linenos">554</span></a><span class="sd"> Args:</span>
</span><span id="L-555"><a href="#L-555"><span class="linenos">555</span></a><span class="sd"> d: dictionary to update.</span>
</span><span id="L-556"><a href="#L-556"><span class="linenos">556</span></a><span class="sd"> keys: the keys that makeup the path to `value`.</span>
</span><span id="L-557"><a href="#L-557"><span class="linenos">557</span></a><span class="sd"> value: the value to set in the dictionary for the given key path.</span>
</span><span id="L-558"><a href="#L-558"><span class="linenos">558</span></a>
</span><span id="L-559"><a href="#L-559"><span class="linenos">559</span></a> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="L-560"><a href="#L-560"><span class="linenos">560</span></a> <span class="n">d</span><span class="p">[</span><span class="n">keys</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value</span>
</span><span id="L-561"><a href="#L-561"><span class="linenos">561</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="L-562"><a href="#L-562"><span class="linenos">562</span></a>
</span><span id="L-563"><a href="#L-563"><span class="linenos">563</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">d</span>
</span><span id="L-564"><a href="#L-564"><span class="linenos">564</span></a> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
</span><span id="L-565"><a href="#L-565"><span class="linenos">565</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">subd</span><span class="p">:</span>
</span><span id="L-566"><a href="#L-566"><span class="linenos">566</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">subd</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="p">{})</span>
</span><span id="L-567"><a href="#L-567"><span class="linenos">567</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-568"><a href="#L-568"><span class="linenos">568</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">subd</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
</span><span id="L-569"><a href="#L-569"><span class="linenos">569</span></a>
</span><span id="L-570"><a href="#L-570"><span class="linenos">570</span></a> <span class="n">subd</span><span class="p">[</span><span class="n">keys</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value</span>
</span><span id="L-571"><a href="#L-571"><span class="linenos">571</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="L-559"><a href="#L-559"><span class="linenos">559</span></a><span class="sd"> Returns:</span>
</span><span id="L-560"><a href="#L-560"><span class="linenos">560</span></a><span class="sd"> The (possibly) updated dictionary.</span>
</span><span id="L-561"><a href="#L-561"><span class="linenos">561</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-562"><a href="#L-562"><span class="linenos">562</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="L-563"><a href="#L-563"><span class="linenos">563</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="L-564"><a href="#L-564"><span class="linenos">564</span></a>
</span><span id="L-565"><a href="#L-565"><span class="linenos">565</span></a> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="L-566"><a href="#L-566"><span class="linenos">566</span></a> <span class="n">d</span><span class="p">[</span><span class="n">keys</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value</span>
</span><span id="L-567"><a href="#L-567"><span class="linenos">567</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="L-568"><a href="#L-568"><span class="linenos">568</span></a>
</span><span id="L-569"><a href="#L-569"><span class="linenos">569</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">d</span>
</span><span id="L-570"><a href="#L-570"><span class="linenos">570</span></a> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
</span><span id="L-571"><a href="#L-571"><span class="linenos">571</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">subd</span><span class="p">:</span>
</span><span id="L-572"><a href="#L-572"><span class="linenos">572</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">subd</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="p">{})</span>
</span><span id="L-573"><a href="#L-573"><span class="linenos">573</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-574"><a href="#L-574"><span class="linenos">574</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">subd</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
</span><span id="L-575"><a href="#L-575"><span class="linenos">575</span></a>
</span><span id="L-576"><a href="#L-576"><span class="linenos">576</span></a> <span class="n">subd</span><span class="p">[</span><span class="n">keys</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value</span>
</span><span id="L-577"><a href="#L-577"><span class="linenos">577</span></a> <span class="k">return</span> <span class="n">d</span>
</span></pre></div>
@ -1202,7 +1208,7 @@ The added table must have the necessary number of qualifiers in its path to matc
</span><span id="AbstractMappingSchema-175"><a href="#AbstractMappingSchema-175"><span class="linenos">175</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="AbstractMappingSchema-176"><a href="#AbstractMappingSchema-176"><span class="linenos">176</span></a>
</span><span id="AbstractMappingSchema-177"><a href="#AbstractMappingSchema-177"><span class="linenos">177</span></a> <span class="k">if</span> <span class="n">value</span> <span class="o">==</span> <span class="n">TrieResult</span><span class="o">.</span><span class="n">PREFIX</span><span class="p">:</span>
</span><span id="AbstractMappingSchema-178"><a href="#AbstractMappingSchema-178"><span class="linenos">178</span></a> <span class="n">possibilities</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">trie</span><span class="p">,</span> <span class="n">depth</span><span class="o">=</span><span class="n">dict_depth</span><span class="p">(</span><span class="n">trie</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
</span><span id="AbstractMappingSchema-178"><a href="#AbstractMappingSchema-178"><span class="linenos">178</span></a> <span class="n">possibilities</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">trie</span><span class="p">)</span>
</span><span id="AbstractMappingSchema-179"><a href="#AbstractMappingSchema-179"><span class="linenos">179</span></a>
</span><span id="AbstractMappingSchema-180"><a href="#AbstractMappingSchema-180"><span class="linenos">180</span></a> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">possibilities</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="AbstractMappingSchema-181"><a href="#AbstractMappingSchema-181"><span class="linenos">181</span></a> <span class="n">parts</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">possibilities</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
@ -1389,7 +1395,7 @@ The added table must have the necessary number of qualifiers in its path to matc
</span><span id="AbstractMappingSchema.find-175"><a href="#AbstractMappingSchema.find-175"><span class="linenos">175</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="AbstractMappingSchema.find-176"><a href="#AbstractMappingSchema.find-176"><span class="linenos">176</span></a>
</span><span id="AbstractMappingSchema.find-177"><a href="#AbstractMappingSchema.find-177"><span class="linenos">177</span></a> <span class="k">if</span> <span class="n">value</span> <span class="o">==</span> <span class="n">TrieResult</span><span class="o">.</span><span class="n">PREFIX</span><span class="p">:</span>
</span><span id="AbstractMappingSchema.find-178"><a href="#AbstractMappingSchema.find-178"><span class="linenos">178</span></a> <span class="n">possibilities</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">trie</span><span class="p">,</span> <span class="n">depth</span><span class="o">=</span><span class="n">dict_depth</span><span class="p">(</span><span class="n">trie</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
</span><span id="AbstractMappingSchema.find-178"><a href="#AbstractMappingSchema.find-178"><span class="linenos">178</span></a> <span class="n">possibilities</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">trie</span><span class="p">)</span>
</span><span id="AbstractMappingSchema.find-179"><a href="#AbstractMappingSchema.find-179"><span class="linenos">179</span></a>
</span><span id="AbstractMappingSchema.find-180"><a href="#AbstractMappingSchema.find-180"><span class="linenos">180</span></a> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">possibilities</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="AbstractMappingSchema.find-181"><a href="#AbstractMappingSchema.find-181"><span class="linenos">181</span></a> <span class="n">parts</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">possibilities</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
@ -1624,91 +1630,96 @@ The added table must have the necessary number of qualifiers in its path to matc
</span><span id="MappingSchema-363"><a href="#MappingSchema-363"><span class="linenos">363</span></a><span class="sd"> The normalized schema mapping.</span>
</span><span id="MappingSchema-364"><a href="#MappingSchema-364"><span class="linenos">364</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="MappingSchema-365"><a href="#MappingSchema-365"><span class="linenos">365</span></a> <span class="n">normalized_mapping</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="MappingSchema-366"><a href="#MappingSchema-366"><span class="linenos">366</span></a> <span class="n">flattened_schema</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">depth</span><span class="o">=</span><span class="n">dict_depth</span><span class="p">(</span><span class="n">schema</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
</span><span id="MappingSchema-367"><a href="#MappingSchema-367"><span class="linenos">367</span></a>
</span><span id="MappingSchema-368"><a href="#MappingSchema-368"><span class="linenos">368</span></a> <span class="k">for</span> <span class="n">keys</span> <span class="ow">in</span> <span class="n">flattened_schema</span><span class="p">:</span>
</span><span id="MappingSchema-369"><a href="#MappingSchema-369"><span class="linenos">369</span></a> <span class="n">columns</span> <span class="o">=</span> <span class="n">nested_get</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="o">*</span><span class="nb">zip</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">keys</span><span class="p">))</span>
</span><span id="MappingSchema-370"><a href="#MappingSchema-370"><span class="linenos">370</span></a>
</span><span id="MappingSchema-371"><a href="#MappingSchema-371"><span class="linenos">371</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">columns</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="MappingSchema-372"><a href="#MappingSchema-372"><span class="linenos">372</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span>
</span><span id="MappingSchema-373"><a href="#MappingSchema-373"><span class="linenos">373</span></a> <span class="sa">f</span><span class="s2">&quot;Table </span><span class="si">{</span><span class="s1">&#39;.&#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">keys</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span><span class="si">}</span><span class="s2"> must match the schema&#39;s nesting level: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">flattened_schema</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="si">}</span><span class="s2">.&quot;</span>
</span><span id="MappingSchema-374"><a href="#MappingSchema-374"><span class="linenos">374</span></a> <span class="p">)</span>
</span><span id="MappingSchema-375"><a href="#MappingSchema-375"><span class="linenos">375</span></a>
</span><span id="MappingSchema-376"><a href="#MappingSchema-376"><span class="linenos">376</span></a> <span class="n">normalized_keys</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_normalize_name</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">is_table</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">]</span>
</span><span id="MappingSchema-377"><a href="#MappingSchema-377"><span class="linenos">377</span></a> <span class="k">for</span> <span class="n">column_name</span><span class="p">,</span> <span class="n">column_type</span> <span class="ow">in</span> <span class="n">columns</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="MappingSchema-378"><a href="#MappingSchema-378"><span class="linenos">378</span></a> <span class="n">nested_set</span><span class="p">(</span>
</span><span id="MappingSchema-379"><a href="#MappingSchema-379"><span class="linenos">379</span></a> <span class="n">normalized_mapping</span><span class="p">,</span>
</span><span id="MappingSchema-380"><a href="#MappingSchema-380"><span class="linenos">380</span></a> <span class="n">normalized_keys</span> <span class="o">+</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_normalize_name</span><span class="p">(</span><span class="n">column_name</span><span class="p">)],</span>
</span><span id="MappingSchema-381"><a href="#MappingSchema-381"><span class="linenos">381</span></a> <span class="n">column_type</span><span class="p">,</span>
</span><span id="MappingSchema-382"><a href="#MappingSchema-382"><span class="linenos">382</span></a> <span class="p">)</span>
</span><span id="MappingSchema-383"><a href="#MappingSchema-383"><span class="linenos">383</span></a>
</span><span id="MappingSchema-384"><a href="#MappingSchema-384"><span class="linenos">384</span></a> <span class="k">return</span> <span class="n">normalized_mapping</span>
</span><span id="MappingSchema-385"><a href="#MappingSchema-385"><span class="linenos">385</span></a>
</span><span id="MappingSchema-386"><a href="#MappingSchema-386"><span class="linenos">386</span></a> <span class="k">def</span> <span class="nf">_normalize_table</span><span class="p">(</span>
</span><span id="MappingSchema-387"><a href="#MappingSchema-387"><span class="linenos">387</span></a> <span class="bp">self</span><span class="p">,</span>
</span><span id="MappingSchema-388"><a href="#MappingSchema-388"><span class="linenos">388</span></a> <span class="n">table</span><span class="p">:</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span> <span class="o">|</span> <span class="nb">str</span><span class="p">,</span>
</span><span id="MappingSchema-389"><a href="#MappingSchema-389"><span class="linenos">389</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="MappingSchema-390"><a href="#MappingSchema-390"><span class="linenos">390</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="MappingSchema-391"><a href="#MappingSchema-391"><span class="linenos">391</span></a> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">:</span>
</span><span id="MappingSchema-392"><a href="#MappingSchema-392"><span class="linenos">392</span></a> <span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span>
</span><span id="MappingSchema-393"><a href="#MappingSchema-393"><span class="linenos">393</span></a> <span class="n">normalize</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">normalize</span> <span class="k">if</span> <span class="n">normalize</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">normalize</span>
</span><span id="MappingSchema-394"><a href="#MappingSchema-394"><span class="linenos">394</span></a>
</span><span id="MappingSchema-395"><a href="#MappingSchema-395"><span class="linenos">395</span></a> <span class="n">normalized_table</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">maybe_parse</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">into</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="n">normalize</span><span class="p">)</span>
</span><span id="MappingSchema-396"><a href="#MappingSchema-396"><span class="linenos">396</span></a>
</span><span id="MappingSchema-397"><a href="#MappingSchema-397"><span class="linenos">397</span></a> <span class="k">if</span> <span class="n">normalize</span><span class="p">:</span>
</span><span id="MappingSchema-398"><a href="#MappingSchema-398"><span class="linenos">398</span></a> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">exp</span><span class="o">.</span><span class="n">TABLE_PARTS</span><span class="p">:</span>
</span><span id="MappingSchema-399"><a href="#MappingSchema-399"><span class="linenos">399</span></a> <span class="n">value</span> <span class="o">=</span> <span class="n">normalized_table</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span>
</span><span id="MappingSchema-400"><a href="#MappingSchema-400"><span class="linenos">400</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">):</span>
</span><span id="MappingSchema-401"><a href="#MappingSchema-401"><span class="linenos">401</span></a> <span class="n">normalized_table</span><span class="o">.</span><span class="n">set</span><span class="p">(</span>
</span><span id="MappingSchema-402"><a href="#MappingSchema-402"><span class="linenos">402</span></a> <span class="n">arg</span><span class="p">,</span>
</span><span id="MappingSchema-403"><a href="#MappingSchema-403"><span class="linenos">403</span></a> <span class="n">normalize_name</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">is_table</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="n">normalize</span><span class="p">),</span>
</span><span id="MappingSchema-404"><a href="#MappingSchema-404"><span class="linenos">404</span></a> <span class="p">)</span>
</span><span id="MappingSchema-405"><a href="#MappingSchema-405"><span class="linenos">405</span></a>
</span><span id="MappingSchema-406"><a href="#MappingSchema-406"><span class="linenos">406</span></a> <span class="k">return</span> <span class="n">normalized_table</span>
</span><span id="MappingSchema-407"><a href="#MappingSchema-407"><span class="linenos">407</span></a>
</span><span id="MappingSchema-408"><a href="#MappingSchema-408"><span class="linenos">408</span></a> <span class="k">def</span> <span class="nf">_normalize_name</span><span class="p">(</span>
</span><span id="MappingSchema-409"><a href="#MappingSchema-409"><span class="linenos">409</span></a> <span class="bp">self</span><span class="p">,</span>
</span><span id="MappingSchema-410"><a href="#MappingSchema-410"><span class="linenos">410</span></a> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">,</span>
</span><span id="MappingSchema-411"><a href="#MappingSchema-411"><span class="linenos">411</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="MappingSchema-412"><a href="#MappingSchema-412"><span class="linenos">412</span></a> <span class="n">is_table</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="MappingSchema-413"><a href="#MappingSchema-413"><span class="linenos">413</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="MappingSchema-414"><a href="#MappingSchema-414"><span class="linenos">414</span></a> <span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span><span id="MappingSchema-415"><a href="#MappingSchema-415"><span class="linenos">415</span></a> <span class="k">return</span> <span class="n">normalize_name</span><span class="p">(</span>
</span><span id="MappingSchema-416"><a href="#MappingSchema-416"><span class="linenos">416</span></a> <span class="n">name</span><span class="p">,</span>
</span><span id="MappingSchema-417"><a href="#MappingSchema-417"><span class="linenos">417</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="MappingSchema-418"><a href="#MappingSchema-418"><span class="linenos">418</span></a> <span class="n">is_table</span><span class="o">=</span><span class="n">is_table</span><span class="p">,</span>
</span><span id="MappingSchema-419"><a href="#MappingSchema-419"><span class="linenos">419</span></a> <span class="n">normalize</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">normalize</span> <span class="k">if</span> <span class="n">normalize</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">normalize</span><span class="p">,</span>
</span><span id="MappingSchema-420"><a href="#MappingSchema-420"><span class="linenos">420</span></a> <span class="p">)</span><span class="o">.</span><span class="n">name</span>
</span><span id="MappingSchema-421"><a href="#MappingSchema-421"><span class="linenos">421</span></a>
</span><span id="MappingSchema-422"><a href="#MappingSchema-422"><span class="linenos">422</span></a> <span class="k">def</span> <span class="nf">depth</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span><span id="MappingSchema-423"><a href="#MappingSchema-423"><span class="linenos">423</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">empty</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span><span class="p">:</span>
</span><span id="MappingSchema-424"><a href="#MappingSchema-424"><span class="linenos">424</span></a> <span class="c1"># The columns themselves are a mapping, but we don&#39;t want to include those</span>
</span><span id="MappingSchema-425"><a href="#MappingSchema-425"><span class="linenos">425</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">depth</span><span class="p">()</span> <span class="o">-</span> <span class="mi">1</span>
</span><span id="MappingSchema-426"><a href="#MappingSchema-426"><span class="linenos">426</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span>
</span><span id="MappingSchema-427"><a href="#MappingSchema-427"><span class="linenos">427</span></a>
</span><span id="MappingSchema-428"><a href="#MappingSchema-428"><span class="linenos">428</span></a> <span class="k">def</span> <span class="nf">_to_data_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">DataType</span><span class="p">:</span>
</span><span id="MappingSchema-429"><a href="#MappingSchema-429"><span class="linenos">429</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="MappingSchema-430"><a href="#MappingSchema-430"><span class="linenos">430</span></a><span class="sd"> Convert a type represented as a string to the corresponding `sqlglot.exp.DataType` object.</span>
</span><span id="MappingSchema-431"><a href="#MappingSchema-431"><span class="linenos">431</span></a>
</span><span id="MappingSchema-432"><a href="#MappingSchema-432"><span class="linenos">432</span></a><span class="sd"> Args:</span>
</span><span id="MappingSchema-433"><a href="#MappingSchema-433"><span class="linenos">433</span></a><span class="sd"> schema_type: the type we want to convert.</span>
</span><span id="MappingSchema-434"><a href="#MappingSchema-434"><span class="linenos">434</span></a><span class="sd"> dialect: the SQL dialect that will be used to parse `schema_type`, if needed.</span>
</span><span id="MappingSchema-435"><a href="#MappingSchema-435"><span class="linenos">435</span></a>
</span><span id="MappingSchema-436"><a href="#MappingSchema-436"><span class="linenos">436</span></a><span class="sd"> Returns:</span>
</span><span id="MappingSchema-437"><a href="#MappingSchema-437"><span class="linenos">437</span></a><span class="sd"> The resulting expression type.</span>
</span><span id="MappingSchema-438"><a href="#MappingSchema-438"><span class="linenos">438</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="MappingSchema-439"><a href="#MappingSchema-439"><span class="linenos">439</span></a> <span class="k">if</span> <span class="n">schema_type</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">:</span>
</span><span id="MappingSchema-440"><a href="#MappingSchema-440"><span class="linenos">440</span></a> <span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span>
</span><span id="MappingSchema-441"><a href="#MappingSchema-441"><span class="linenos">441</span></a> <span class="n">udt</span> <span class="o">=</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">SUPPORTS_USER_DEFINED_TYPES</span>
</span><span id="MappingSchema-442"><a href="#MappingSchema-442"><span class="linenos">442</span></a>
</span><span id="MappingSchema-443"><a href="#MappingSchema-443"><span class="linenos">443</span></a> <span class="k">try</span><span class="p">:</span>
</span><span id="MappingSchema-444"><a href="#MappingSchema-444"><span class="linenos">444</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">DataType</span><span class="o">.</span><span class="n">build</span><span class="p">(</span><span class="n">schema_type</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">udt</span><span class="o">=</span><span class="n">udt</span><span class="p">)</span>
</span><span id="MappingSchema-445"><a href="#MappingSchema-445"><span class="linenos">445</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">[</span><span class="n">schema_type</span><span class="p">]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="MappingSchema-446"><a href="#MappingSchema-446"><span class="linenos">446</span></a> <span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
</span><span id="MappingSchema-447"><a href="#MappingSchema-447"><span class="linenos">447</span></a> <span class="n">in_dialect</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot; in dialect </span><span class="si">{</span><span class="n">dialect</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">if</span> <span class="n">dialect</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
</span><span id="MappingSchema-448"><a href="#MappingSchema-448"><span class="linenos">448</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to build type &#39;</span><span class="si">{</span><span class="n">schema_type</span><span class="si">}</span><span class="s2">&#39;</span><span class="si">{</span><span class="n">in_dialect</span><span class="si">}</span><span class="s2">.&quot;</span><span class="p">)</span>
</span><span id="MappingSchema-449"><a href="#MappingSchema-449"><span class="linenos">449</span></a>
</span><span id="MappingSchema-450"><a href="#MappingSchema-450"><span class="linenos">450</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">[</span><span class="n">schema_type</span><span class="p">]</span>
</span><span id="MappingSchema-366"><a href="#MappingSchema-366"><span class="linenos">366</span></a> <span class="n">flattened_schema</span> <span class="o">=</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">)</span>
</span><span id="MappingSchema-367"><a href="#MappingSchema-367"><span class="linenos">367</span></a> <span class="n">error_msg</span> <span class="o">=</span> <span class="s2">&quot;Table </span><span class="si">{}</span><span class="s2"> must match the schema&#39;s nesting level: </span><span class="si">{}</span><span class="s2">.&quot;</span>
</span><span id="MappingSchema-368"><a href="#MappingSchema-368"><span class="linenos">368</span></a>
</span><span id="MappingSchema-369"><a href="#MappingSchema-369"><span class="linenos">369</span></a> <span class="k">for</span> <span class="n">keys</span> <span class="ow">in</span> <span class="n">flattened_schema</span><span class="p">:</span>
</span><span id="MappingSchema-370"><a href="#MappingSchema-370"><span class="linenos">370</span></a> <span class="n">columns</span> <span class="o">=</span> <span class="n">nested_get</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="o">*</span><span class="nb">zip</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">keys</span><span class="p">))</span>
</span><span id="MappingSchema-371"><a href="#MappingSchema-371"><span class="linenos">371</span></a>
</span><span id="MappingSchema-372"><a href="#MappingSchema-372"><span class="linenos">372</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">columns</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="MappingSchema-373"><a href="#MappingSchema-373"><span class="linenos">373</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span><span class="n">error_msg</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">keys</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]),</span> <span class="nb">len</span><span class="p">(</span><span class="n">flattened_schema</span><span class="p">[</span><span class="mi">0</span><span class="p">])))</span>
</span><span id="MappingSchema-374"><a href="#MappingSchema-374"><span class="linenos">374</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">first</span><span class="p">(</span><span class="n">columns</span><span class="o">.</span><span class="n">values</span><span class="p">()),</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="MappingSchema-375"><a href="#MappingSchema-375"><span class="linenos">375</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span>
</span><span id="MappingSchema-376"><a href="#MappingSchema-376"><span class="linenos">376</span></a> <span class="n">error_msg</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
</span><span id="MappingSchema-377"><a href="#MappingSchema-377"><span class="linenos">377</span></a> <span class="s2">&quot;.&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">keys</span> <span class="o">+</span> <span class="n">flatten_schema</span><span class="p">(</span><span class="n">columns</span><span class="p">)[</span><span class="mi">0</span><span class="p">]),</span> <span class="nb">len</span><span class="p">(</span><span class="n">flattened_schema</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</span><span id="MappingSchema-378"><a href="#MappingSchema-378"><span class="linenos">378</span></a> <span class="p">),</span>
</span><span id="MappingSchema-379"><a href="#MappingSchema-379"><span class="linenos">379</span></a> <span class="p">)</span>
</span><span id="MappingSchema-380"><a href="#MappingSchema-380"><span class="linenos">380</span></a>
</span><span id="MappingSchema-381"><a href="#MappingSchema-381"><span class="linenos">381</span></a> <span class="n">normalized_keys</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_normalize_name</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">is_table</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">]</span>
</span><span id="MappingSchema-382"><a href="#MappingSchema-382"><span class="linenos">382</span></a> <span class="k">for</span> <span class="n">column_name</span><span class="p">,</span> <span class="n">column_type</span> <span class="ow">in</span> <span class="n">columns</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="MappingSchema-383"><a href="#MappingSchema-383"><span class="linenos">383</span></a> <span class="n">nested_set</span><span class="p">(</span>
</span><span id="MappingSchema-384"><a href="#MappingSchema-384"><span class="linenos">384</span></a> <span class="n">normalized_mapping</span><span class="p">,</span>
</span><span id="MappingSchema-385"><a href="#MappingSchema-385"><span class="linenos">385</span></a> <span class="n">normalized_keys</span> <span class="o">+</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_normalize_name</span><span class="p">(</span><span class="n">column_name</span><span class="p">)],</span>
</span><span id="MappingSchema-386"><a href="#MappingSchema-386"><span class="linenos">386</span></a> <span class="n">column_type</span><span class="p">,</span>
</span><span id="MappingSchema-387"><a href="#MappingSchema-387"><span class="linenos">387</span></a> <span class="p">)</span>
</span><span id="MappingSchema-388"><a href="#MappingSchema-388"><span class="linenos">388</span></a>
</span><span id="MappingSchema-389"><a href="#MappingSchema-389"><span class="linenos">389</span></a> <span class="k">return</span> <span class="n">normalized_mapping</span>
</span><span id="MappingSchema-390"><a href="#MappingSchema-390"><span class="linenos">390</span></a>
</span><span id="MappingSchema-391"><a href="#MappingSchema-391"><span class="linenos">391</span></a> <span class="k">def</span> <span class="nf">_normalize_table</span><span class="p">(</span>
</span><span id="MappingSchema-392"><a href="#MappingSchema-392"><span class="linenos">392</span></a> <span class="bp">self</span><span class="p">,</span>
</span><span id="MappingSchema-393"><a href="#MappingSchema-393"><span class="linenos">393</span></a> <span class="n">table</span><span class="p">:</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span> <span class="o">|</span> <span class="nb">str</span><span class="p">,</span>
</span><span id="MappingSchema-394"><a href="#MappingSchema-394"><span class="linenos">394</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="MappingSchema-395"><a href="#MappingSchema-395"><span class="linenos">395</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="MappingSchema-396"><a href="#MappingSchema-396"><span class="linenos">396</span></a> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">:</span>
</span><span id="MappingSchema-397"><a href="#MappingSchema-397"><span class="linenos">397</span></a> <span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span>
</span><span id="MappingSchema-398"><a href="#MappingSchema-398"><span class="linenos">398</span></a> <span class="n">normalize</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">normalize</span> <span class="k">if</span> <span class="n">normalize</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">normalize</span>
</span><span id="MappingSchema-399"><a href="#MappingSchema-399"><span class="linenos">399</span></a>
</span><span id="MappingSchema-400"><a href="#MappingSchema-400"><span class="linenos">400</span></a> <span class="n">normalized_table</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">maybe_parse</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">into</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="n">normalize</span><span class="p">)</span>
</span><span id="MappingSchema-401"><a href="#MappingSchema-401"><span class="linenos">401</span></a>
</span><span id="MappingSchema-402"><a href="#MappingSchema-402"><span class="linenos">402</span></a> <span class="k">if</span> <span class="n">normalize</span><span class="p">:</span>
</span><span id="MappingSchema-403"><a href="#MappingSchema-403"><span class="linenos">403</span></a> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">exp</span><span class="o">.</span><span class="n">TABLE_PARTS</span><span class="p">:</span>
</span><span id="MappingSchema-404"><a href="#MappingSchema-404"><span class="linenos">404</span></a> <span class="n">value</span> <span class="o">=</span> <span class="n">normalized_table</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span>
</span><span id="MappingSchema-405"><a href="#MappingSchema-405"><span class="linenos">405</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">):</span>
</span><span id="MappingSchema-406"><a href="#MappingSchema-406"><span class="linenos">406</span></a> <span class="n">normalized_table</span><span class="o">.</span><span class="n">set</span><span class="p">(</span>
</span><span id="MappingSchema-407"><a href="#MappingSchema-407"><span class="linenos">407</span></a> <span class="n">arg</span><span class="p">,</span>
</span><span id="MappingSchema-408"><a href="#MappingSchema-408"><span class="linenos">408</span></a> <span class="n">normalize_name</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">is_table</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="n">normalize</span><span class="p">),</span>
</span><span id="MappingSchema-409"><a href="#MappingSchema-409"><span class="linenos">409</span></a> <span class="p">)</span>
</span><span id="MappingSchema-410"><a href="#MappingSchema-410"><span class="linenos">410</span></a>
</span><span id="MappingSchema-411"><a href="#MappingSchema-411"><span class="linenos">411</span></a> <span class="k">return</span> <span class="n">normalized_table</span>
</span><span id="MappingSchema-412"><a href="#MappingSchema-412"><span class="linenos">412</span></a>
</span><span id="MappingSchema-413"><a href="#MappingSchema-413"><span class="linenos">413</span></a> <span class="k">def</span> <span class="nf">_normalize_name</span><span class="p">(</span>
</span><span id="MappingSchema-414"><a href="#MappingSchema-414"><span class="linenos">414</span></a> <span class="bp">self</span><span class="p">,</span>
</span><span id="MappingSchema-415"><a href="#MappingSchema-415"><span class="linenos">415</span></a> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">,</span>
</span><span id="MappingSchema-416"><a href="#MappingSchema-416"><span class="linenos">416</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="MappingSchema-417"><a href="#MappingSchema-417"><span class="linenos">417</span></a> <span class="n">is_table</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="MappingSchema-418"><a href="#MappingSchema-418"><span class="linenos">418</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="MappingSchema-419"><a href="#MappingSchema-419"><span class="linenos">419</span></a> <span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span><span id="MappingSchema-420"><a href="#MappingSchema-420"><span class="linenos">420</span></a> <span class="k">return</span> <span class="n">normalize_name</span><span class="p">(</span>
</span><span id="MappingSchema-421"><a href="#MappingSchema-421"><span class="linenos">421</span></a> <span class="n">name</span><span class="p">,</span>
</span><span id="MappingSchema-422"><a href="#MappingSchema-422"><span class="linenos">422</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="MappingSchema-423"><a href="#MappingSchema-423"><span class="linenos">423</span></a> <span class="n">is_table</span><span class="o">=</span><span class="n">is_table</span><span class="p">,</span>
</span><span id="MappingSchema-424"><a href="#MappingSchema-424"><span class="linenos">424</span></a> <span class="n">normalize</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">normalize</span> <span class="k">if</span> <span class="n">normalize</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">normalize</span><span class="p">,</span>
</span><span id="MappingSchema-425"><a href="#MappingSchema-425"><span class="linenos">425</span></a> <span class="p">)</span><span class="o">.</span><span class="n">name</span>
</span><span id="MappingSchema-426"><a href="#MappingSchema-426"><span class="linenos">426</span></a>
</span><span id="MappingSchema-427"><a href="#MappingSchema-427"><span class="linenos">427</span></a> <span class="k">def</span> <span class="nf">depth</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span><span id="MappingSchema-428"><a href="#MappingSchema-428"><span class="linenos">428</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">empty</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span><span class="p">:</span>
</span><span id="MappingSchema-429"><a href="#MappingSchema-429"><span class="linenos">429</span></a> <span class="c1"># The columns themselves are a mapping, but we don&#39;t want to include those</span>
</span><span id="MappingSchema-430"><a href="#MappingSchema-430"><span class="linenos">430</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">depth</span><span class="p">()</span> <span class="o">-</span> <span class="mi">1</span>
</span><span id="MappingSchema-431"><a href="#MappingSchema-431"><span class="linenos">431</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span>
</span><span id="MappingSchema-432"><a href="#MappingSchema-432"><span class="linenos">432</span></a>
</span><span id="MappingSchema-433"><a href="#MappingSchema-433"><span class="linenos">433</span></a> <span class="k">def</span> <span class="nf">_to_data_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">DataType</span><span class="p">:</span>
</span><span id="MappingSchema-434"><a href="#MappingSchema-434"><span class="linenos">434</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="MappingSchema-435"><a href="#MappingSchema-435"><span class="linenos">435</span></a><span class="sd"> Convert a type represented as a string to the corresponding `sqlglot.exp.DataType` object.</span>
</span><span id="MappingSchema-436"><a href="#MappingSchema-436"><span class="linenos">436</span></a>
</span><span id="MappingSchema-437"><a href="#MappingSchema-437"><span class="linenos">437</span></a><span class="sd"> Args:</span>
</span><span id="MappingSchema-438"><a href="#MappingSchema-438"><span class="linenos">438</span></a><span class="sd"> schema_type: the type we want to convert.</span>
</span><span id="MappingSchema-439"><a href="#MappingSchema-439"><span class="linenos">439</span></a><span class="sd"> dialect: the SQL dialect that will be used to parse `schema_type`, if needed.</span>
</span><span id="MappingSchema-440"><a href="#MappingSchema-440"><span class="linenos">440</span></a>
</span><span id="MappingSchema-441"><a href="#MappingSchema-441"><span class="linenos">441</span></a><span class="sd"> Returns:</span>
</span><span id="MappingSchema-442"><a href="#MappingSchema-442"><span class="linenos">442</span></a><span class="sd"> The resulting expression type.</span>
</span><span id="MappingSchema-443"><a href="#MappingSchema-443"><span class="linenos">443</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="MappingSchema-444"><a href="#MappingSchema-444"><span class="linenos">444</span></a> <span class="k">if</span> <span class="n">schema_type</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">:</span>
</span><span id="MappingSchema-445"><a href="#MappingSchema-445"><span class="linenos">445</span></a> <span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span>
</span><span id="MappingSchema-446"><a href="#MappingSchema-446"><span class="linenos">446</span></a> <span class="n">udt</span> <span class="o">=</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">SUPPORTS_USER_DEFINED_TYPES</span>
</span><span id="MappingSchema-447"><a href="#MappingSchema-447"><span class="linenos">447</span></a>
</span><span id="MappingSchema-448"><a href="#MappingSchema-448"><span class="linenos">448</span></a> <span class="k">try</span><span class="p">:</span>
</span><span id="MappingSchema-449"><a href="#MappingSchema-449"><span class="linenos">449</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">DataType</span><span class="o">.</span><span class="n">build</span><span class="p">(</span><span class="n">schema_type</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">udt</span><span class="o">=</span><span class="n">udt</span><span class="p">)</span>
</span><span id="MappingSchema-450"><a href="#MappingSchema-450"><span class="linenos">450</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">[</span><span class="n">schema_type</span><span class="p">]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="MappingSchema-451"><a href="#MappingSchema-451"><span class="linenos">451</span></a> <span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
</span><span id="MappingSchema-452"><a href="#MappingSchema-452"><span class="linenos">452</span></a> <span class="n">in_dialect</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot; in dialect </span><span class="si">{</span><span class="n">dialect</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">if</span> <span class="n">dialect</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
</span><span id="MappingSchema-453"><a href="#MappingSchema-453"><span class="linenos">453</span></a> <span class="k">raise</span> <span class="n">SchemaError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to build type &#39;</span><span class="si">{</span><span class="n">schema_type</span><span class="si">}</span><span class="s2">&#39;</span><span class="si">{</span><span class="n">in_dialect</span><span class="si">}</span><span class="s2">.&quot;</span><span class="p">)</span>
</span><span id="MappingSchema-454"><a href="#MappingSchema-454"><span class="linenos">454</span></a>
</span><span id="MappingSchema-455"><a href="#MappingSchema-455"><span class="linenos">455</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_type_mapping_cache</span><span class="p">[</span><span class="n">schema_type</span><span class="p">]</span>
</span></pre></div>
@ -2093,11 +2104,11 @@ The added table must have the necessary number of qualifiers in its path to matc
</div>
<a class="headerlink" href="#MappingSchema.depth"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="MappingSchema.depth-422"><a href="#MappingSchema.depth-422"><span class="linenos">422</span></a> <span class="k">def</span> <span class="nf">depth</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span><span id="MappingSchema.depth-423"><a href="#MappingSchema.depth-423"><span class="linenos">423</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">empty</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span><span class="p">:</span>
</span><span id="MappingSchema.depth-424"><a href="#MappingSchema.depth-424"><span class="linenos">424</span></a> <span class="c1"># The columns themselves are a mapping, but we don&#39;t want to include those</span>
</span><span id="MappingSchema.depth-425"><a href="#MappingSchema.depth-425"><span class="linenos">425</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">depth</span><span class="p">()</span> <span class="o">-</span> <span class="mi">1</span>
</span><span id="MappingSchema.depth-426"><a href="#MappingSchema.depth-426"><span class="linenos">426</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="MappingSchema.depth-427"><a href="#MappingSchema.depth-427"><span class="linenos">427</span></a> <span class="k">def</span> <span class="nf">depth</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span><span id="MappingSchema.depth-428"><a href="#MappingSchema.depth-428"><span class="linenos">428</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">empty</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span><span class="p">:</span>
</span><span id="MappingSchema.depth-429"><a href="#MappingSchema.depth-429"><span class="linenos">429</span></a> <span class="c1"># The columns themselves are a mapping, but we don&#39;t want to include those</span>
</span><span id="MappingSchema.depth-430"><a href="#MappingSchema.depth-430"><span class="linenos">430</span></a> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">depth</span><span class="p">()</span> <span class="o">-</span> <span class="mi">1</span>
</span><span id="MappingSchema.depth-431"><a href="#MappingSchema.depth-431"><span class="linenos">431</span></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_depth</span>
</span></pre></div>
@ -2131,21 +2142,21 @@ The added table must have the necessary number of qualifiers in its path to matc
</div>
<a class="headerlink" href="#normalize_name"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="normalize_name-453"><a href="#normalize_name-453"><span class="linenos">453</span></a><span class="k">def</span> <span class="nf">normalize_name</span><span class="p">(</span>
</span><span id="normalize_name-454"><a href="#normalize_name-454"><span class="linenos">454</span></a> <span class="n">identifier</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">,</span>
</span><span id="normalize_name-455"><a href="#normalize_name-455"><span class="linenos">455</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="normalize_name-456"><a href="#normalize_name-456"><span class="linenos">456</span></a> <span class="n">is_table</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="normalize_name-457"><a href="#normalize_name-457"><span class="linenos">457</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="normalize_name-458"><a href="#normalize_name-458"><span class="linenos">458</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">:</span>
</span><span id="normalize_name-459"><a href="#normalize_name-459"><span class="linenos">459</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">identifier</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span><span id="normalize_name-460"><a href="#normalize_name-460"><span class="linenos">460</span></a> <span class="n">identifier</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">parse_identifier</span><span class="p">(</span><span class="n">identifier</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="normalize_name-461"><a href="#normalize_name-461"><span class="linenos">461</span></a>
</span><span id="normalize_name-462"><a href="#normalize_name-462"><span class="linenos">462</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">normalize</span><span class="p">:</span>
</span><span id="normalize_name-463"><a href="#normalize_name-463"><span class="linenos">463</span></a> <span class="k">return</span> <span class="n">identifier</span>
</span><span id="normalize_name-464"><a href="#normalize_name-464"><span class="linenos">464</span></a>
</span><span id="normalize_name-465"><a href="#normalize_name-465"><span class="linenos">465</span></a> <span class="c1"># this is used for normalize_identifier, bigquery has special rules pertaining tables</span>
</span><span id="normalize_name-466"><a href="#normalize_name-466"><span class="linenos">466</span></a> <span class="n">identifier</span><span class="o">.</span><span class="n">meta</span><span class="p">[</span><span class="s2">&quot;is_table&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">is_table</span>
</span><span id="normalize_name-467"><a href="#normalize_name-467"><span class="linenos">467</span></a> <span class="k">return</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">normalize_identifier</span><span class="p">(</span><span class="n">identifier</span><span class="p">)</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="normalize_name-458"><a href="#normalize_name-458"><span class="linenos">458</span></a><span class="k">def</span> <span class="nf">normalize_name</span><span class="p">(</span>
</span><span id="normalize_name-459"><a href="#normalize_name-459"><span class="linenos">459</span></a> <span class="n">identifier</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">,</span>
</span><span id="normalize_name-460"><a href="#normalize_name-460"><span class="linenos">460</span></a> <span class="n">dialect</span><span class="p">:</span> <span class="n">DialectType</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="normalize_name-461"><a href="#normalize_name-461"><span class="linenos">461</span></a> <span class="n">is_table</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="normalize_name-462"><a href="#normalize_name-462"><span class="linenos">462</span></a> <span class="n">normalize</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="normalize_name-463"><a href="#normalize_name-463"><span class="linenos">463</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Identifier</span><span class="p">:</span>
</span><span id="normalize_name-464"><a href="#normalize_name-464"><span class="linenos">464</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">identifier</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span><span id="normalize_name-465"><a href="#normalize_name-465"><span class="linenos">465</span></a> <span class="n">identifier</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">parse_identifier</span><span class="p">(</span><span class="n">identifier</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="normalize_name-466"><a href="#normalize_name-466"><span class="linenos">466</span></a>
</span><span id="normalize_name-467"><a href="#normalize_name-467"><span class="linenos">467</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">normalize</span><span class="p">:</span>
</span><span id="normalize_name-468"><a href="#normalize_name-468"><span class="linenos">468</span></a> <span class="k">return</span> <span class="n">identifier</span>
</span><span id="normalize_name-469"><a href="#normalize_name-469"><span class="linenos">469</span></a>
</span><span id="normalize_name-470"><a href="#normalize_name-470"><span class="linenos">470</span></a> <span class="c1"># this is used for normalize_identifier, bigquery has special rules pertaining tables</span>
</span><span id="normalize_name-471"><a href="#normalize_name-471"><span class="linenos">471</span></a> <span class="n">identifier</span><span class="o">.</span><span class="n">meta</span><span class="p">[</span><span class="s2">&quot;is_table&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">is_table</span>
</span><span id="normalize_name-472"><a href="#normalize_name-472"><span class="linenos">472</span></a> <span class="k">return</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">normalize_identifier</span><span class="p">(</span><span class="n">identifier</span><span class="p">)</span>
</span></pre></div>
@ -2163,11 +2174,11 @@ The added table must have the necessary number of qualifiers in its path to matc
</div>
<a class="headerlink" href="#ensure_schema"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="ensure_schema-470"><a href="#ensure_schema-470"><span class="linenos">470</span></a><span class="k">def</span> <span class="nf">ensure_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">:</span> <span class="n">Schema</span> <span class="o">|</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">],</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Schema</span><span class="p">:</span>
</span><span id="ensure_schema-471"><a href="#ensure_schema-471"><span class="linenos">471</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">Schema</span><span class="p">):</span>
</span><span id="ensure_schema-472"><a href="#ensure_schema-472"><span class="linenos">472</span></a> <span class="k">return</span> <span class="n">schema</span>
</span><span id="ensure_schema-473"><a href="#ensure_schema-473"><span class="linenos">473</span></a>
</span><span id="ensure_schema-474"><a href="#ensure_schema-474"><span class="linenos">474</span></a> <span class="k">return</span> <span class="n">MappingSchema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="ensure_schema-475"><a href="#ensure_schema-475"><span class="linenos">475</span></a><span class="k">def</span> <span class="nf">ensure_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">:</span> <span class="n">Schema</span> <span class="o">|</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">],</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Schema</span><span class="p">:</span>
</span><span id="ensure_schema-476"><a href="#ensure_schema-476"><span class="linenos">476</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">Schema</span><span class="p">):</span>
</span><span id="ensure_schema-477"><a href="#ensure_schema-477"><span class="linenos">477</span></a> <span class="k">return</span> <span class="n">schema</span>
</span><span id="ensure_schema-478"><a href="#ensure_schema-478"><span class="linenos">478</span></a>
</span><span id="ensure_schema-479"><a href="#ensure_schema-479"><span class="linenos">479</span></a> <span class="k">return</span> <span class="n">MappingSchema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span></pre></div>
@ -2185,24 +2196,24 @@ The added table must have the necessary number of qualifiers in its path to matc
</div>
<a class="headerlink" href="#ensure_column_mapping"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="ensure_column_mapping-477"><a href="#ensure_column_mapping-477"><span class="linenos">477</span></a><span class="k">def</span> <span class="nf">ensure_column_mapping</span><span class="p">(</span><span class="n">mapping</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">ColumnMapping</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">:</span>
</span><span id="ensure_column_mapping-478"><a href="#ensure_column_mapping-478"><span class="linenos">478</span></a> <span class="k">if</span> <span class="n">mapping</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span><span id="ensure_column_mapping-479"><a href="#ensure_column_mapping-479"><span class="linenos">479</span></a> <span class="k">return</span> <span class="p">{}</span>
</span><span id="ensure_column_mapping-480"><a href="#ensure_column_mapping-480"><span class="linenos">480</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="ensure_column_mapping-481"><a href="#ensure_column_mapping-481"><span class="linenos">481</span></a> <span class="k">return</span> <span class="n">mapping</span>
</span><span id="ensure_column_mapping-482"><a href="#ensure_column_mapping-482"><span class="linenos">482</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span><span id="ensure_column_mapping-483"><a href="#ensure_column_mapping-483"><span class="linenos">483</span></a> <span class="n">col_name_type_strs</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">mapping</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
</span><span id="ensure_column_mapping-484"><a href="#ensure_column_mapping-484"><span class="linenos">484</span></a> <span class="k">return</span> <span class="p">{</span>
</span><span id="ensure_column_mapping-485"><a href="#ensure_column_mapping-485"><span class="linenos">485</span></a> <span class="n">name_type_str</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span> <span class="n">name_type_str</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span><span id="ensure_column_mapping-486"><a href="#ensure_column_mapping-486"><span class="linenos">486</span></a> <span class="k">for</span> <span class="n">name_type_str</span> <span class="ow">in</span> <span class="n">col_name_type_strs</span>
</span><span id="ensure_column_mapping-487"><a href="#ensure_column_mapping-487"><span class="linenos">487</span></a> <span class="p">}</span>
</span><span id="ensure_column_mapping-488"><a href="#ensure_column_mapping-488"><span class="linenos">488</span></a> <span class="c1"># Check if mapping looks like a DataFrame StructType</span>
</span><span id="ensure_column_mapping-489"><a href="#ensure_column_mapping-489"><span class="linenos">489</span></a> <span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="s2">&quot;simpleString&quot;</span><span class="p">):</span>
</span><span id="ensure_column_mapping-490"><a href="#ensure_column_mapping-490"><span class="linenos">490</span></a> <span class="k">return</span> <span class="p">{</span><span class="n">struct_field</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">struct_field</span><span class="o">.</span><span class="n">dataType</span><span class="o">.</span><span class="n">simpleString</span><span class="p">()</span> <span class="k">for</span> <span class="n">struct_field</span> <span class="ow">in</span> <span class="n">mapping</span><span class="p">}</span>
</span><span id="ensure_column_mapping-491"><a href="#ensure_column_mapping-491"><span class="linenos">491</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
</span><span id="ensure_column_mapping-492"><a href="#ensure_column_mapping-492"><span class="linenos">492</span></a> <span class="k">return</span> <span class="p">{</span><span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span> <span class="kc">None</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">mapping</span><span class="p">}</span>
</span><span id="ensure_column_mapping-493"><a href="#ensure_column_mapping-493"><span class="linenos">493</span></a>
</span><span id="ensure_column_mapping-494"><a href="#ensure_column_mapping-494"><span class="linenos">494</span></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid mapping provided: </span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="n">mapping</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="ensure_column_mapping-482"><a href="#ensure_column_mapping-482"><span class="linenos">482</span></a><span class="k">def</span> <span class="nf">ensure_column_mapping</span><span class="p">(</span><span class="n">mapping</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">ColumnMapping</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">:</span>
</span><span id="ensure_column_mapping-483"><a href="#ensure_column_mapping-483"><span class="linenos">483</span></a> <span class="k">if</span> <span class="n">mapping</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span><span id="ensure_column_mapping-484"><a href="#ensure_column_mapping-484"><span class="linenos">484</span></a> <span class="k">return</span> <span class="p">{}</span>
</span><span id="ensure_column_mapping-485"><a href="#ensure_column_mapping-485"><span class="linenos">485</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="ensure_column_mapping-486"><a href="#ensure_column_mapping-486"><span class="linenos">486</span></a> <span class="k">return</span> <span class="n">mapping</span>
</span><span id="ensure_column_mapping-487"><a href="#ensure_column_mapping-487"><span class="linenos">487</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span><span id="ensure_column_mapping-488"><a href="#ensure_column_mapping-488"><span class="linenos">488</span></a> <span class="n">col_name_type_strs</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">mapping</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
</span><span id="ensure_column_mapping-489"><a href="#ensure_column_mapping-489"><span class="linenos">489</span></a> <span class="k">return</span> <span class="p">{</span>
</span><span id="ensure_column_mapping-490"><a href="#ensure_column_mapping-490"><span class="linenos">490</span></a> <span class="n">name_type_str</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span> <span class="n">name_type_str</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span><span id="ensure_column_mapping-491"><a href="#ensure_column_mapping-491"><span class="linenos">491</span></a> <span class="k">for</span> <span class="n">name_type_str</span> <span class="ow">in</span> <span class="n">col_name_type_strs</span>
</span><span id="ensure_column_mapping-492"><a href="#ensure_column_mapping-492"><span class="linenos">492</span></a> <span class="p">}</span>
</span><span id="ensure_column_mapping-493"><a href="#ensure_column_mapping-493"><span class="linenos">493</span></a> <span class="c1"># Check if mapping looks like a DataFrame StructType</span>
</span><span id="ensure_column_mapping-494"><a href="#ensure_column_mapping-494"><span class="linenos">494</span></a> <span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="s2">&quot;simpleString&quot;</span><span class="p">):</span>
</span><span id="ensure_column_mapping-495"><a href="#ensure_column_mapping-495"><span class="linenos">495</span></a> <span class="k">return</span> <span class="p">{</span><span class="n">struct_field</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">struct_field</span><span class="o">.</span><span class="n">dataType</span><span class="o">.</span><span class="n">simpleString</span><span class="p">()</span> <span class="k">for</span> <span class="n">struct_field</span> <span class="ow">in</span> <span class="n">mapping</span><span class="p">}</span>
</span><span id="ensure_column_mapping-496"><a href="#ensure_column_mapping-496"><span class="linenos">496</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">mapping</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
</span><span id="ensure_column_mapping-497"><a href="#ensure_column_mapping-497"><span class="linenos">497</span></a> <span class="k">return</span> <span class="p">{</span><span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span> <span class="kc">None</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">mapping</span><span class="p">}</span>
</span><span id="ensure_column_mapping-498"><a href="#ensure_column_mapping-498"><span class="linenos">498</span></a>
</span><span id="ensure_column_mapping-499"><a href="#ensure_column_mapping-499"><span class="linenos">499</span></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid mapping provided: </span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="n">mapping</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span></pre></div>
@ -2214,25 +2225,26 @@ The added table must have the necessary number of qualifiers in its path to matc
<div class="attr function">
<span class="def">def</span>
<span class="name">flatten_schema</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">schema</span><span class="p">:</span> <span class="n">Dict</span>,</span><span class="param"> <span class="n">depth</span><span class="p">:</span> <span class="nb">int</span>,</span><span class="param"> <span class="n">keys</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span></span><span class="return-annotation">) -> <span class="n">List</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span>:</span></span>
<span class="name">flatten_schema</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">schema</span><span class="p">:</span> <span class="n">Dict</span>,</span><span class="param"> <span class="n">depth</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">keys</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span></span><span class="return-annotation">) -> <span class="n">List</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span>:</span></span>
<label class="view-source-button" for="flatten_schema-view-source"><span>View Source</span></label>
</div>
<a class="headerlink" href="#flatten_schema"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="flatten_schema-497"><a href="#flatten_schema-497"><span class="linenos">497</span></a><span class="k">def</span> <span class="nf">flatten_schema</span><span class="p">(</span>
</span><span id="flatten_schema-498"><a href="#flatten_schema-498"><span class="linenos">498</span></a> <span class="n">schema</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">depth</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">keys</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="flatten_schema-499"><a href="#flatten_schema-499"><span class="linenos">499</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]:</span>
</span><span id="flatten_schema-500"><a href="#flatten_schema-500"><span class="linenos">500</span></a> <span class="n">tables</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="flatten_schema-501"><a href="#flatten_schema-501"><span class="linenos">501</span></a> <span class="n">keys</span> <span class="o">=</span> <span class="n">keys</span> <span class="ow">or</span> <span class="p">[]</span>
</span><span id="flatten_schema-502"><a href="#flatten_schema-502"><span class="linenos">502</span></a>
</span><span id="flatten_schema-503"><a href="#flatten_schema-503"><span class="linenos">503</span></a> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">schema</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="flatten_schema-504"><a href="#flatten_schema-504"><span class="linenos">504</span></a> <span class="k">if</span> <span class="n">depth</span> <span class="o">&gt;=</span> <span class="mi">2</span><span class="p">:</span>
</span><span id="flatten_schema-505"><a href="#flatten_schema-505"><span class="linenos">505</span></a> <span class="n">tables</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">flatten_schema</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">depth</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="n">keys</span> <span class="o">+</span> <span class="p">[</span><span class="n">k</span><span class="p">]))</span>
</span><span id="flatten_schema-506"><a href="#flatten_schema-506"><span class="linenos">506</span></a> <span class="k">elif</span> <span class="n">depth</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="flatten_schema-507"><a href="#flatten_schema-507"><span class="linenos">507</span></a> <span class="n">tables</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">keys</span> <span class="o">+</span> <span class="p">[</span><span class="n">k</span><span class="p">])</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="flatten_schema-502"><a href="#flatten_schema-502"><span class="linenos">502</span></a><span class="k">def</span> <span class="nf">flatten_schema</span><span class="p">(</span>
</span><span id="flatten_schema-503"><a href="#flatten_schema-503"><span class="linenos">503</span></a> <span class="n">schema</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">depth</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">keys</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="flatten_schema-504"><a href="#flatten_schema-504"><span class="linenos">504</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]:</span>
</span><span id="flatten_schema-505"><a href="#flatten_schema-505"><span class="linenos">505</span></a> <span class="n">tables</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="flatten_schema-506"><a href="#flatten_schema-506"><span class="linenos">506</span></a> <span class="n">keys</span> <span class="o">=</span> <span class="n">keys</span> <span class="ow">or</span> <span class="p">[]</span>
</span><span id="flatten_schema-507"><a href="#flatten_schema-507"><span class="linenos">507</span></a> <span class="n">depth</span> <span class="o">=</span> <span class="n">dict_depth</span><span class="p">(</span><span class="n">schema</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">depth</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">depth</span>
</span><span id="flatten_schema-508"><a href="#flatten_schema-508"><span class="linenos">508</span></a>
</span><span id="flatten_schema-509"><a href="#flatten_schema-509"><span class="linenos">509</span></a> <span class="k">return</span> <span class="n">tables</span>
</span><span id="flatten_schema-509"><a href="#flatten_schema-509"><span class="linenos">509</span></a> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">schema</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span><span id="flatten_schema-510"><a href="#flatten_schema-510"><span class="linenos">510</span></a> <span class="k">if</span> <span class="n">depth</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span><span id="flatten_schema-511"><a href="#flatten_schema-511"><span class="linenos">511</span></a> <span class="n">tables</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">keys</span> <span class="o">+</span> <span class="p">[</span><span class="n">k</span><span class="p">])</span>
</span><span id="flatten_schema-512"><a href="#flatten_schema-512"><span class="linenos">512</span></a> <span class="k">elif</span> <span class="n">depth</span> <span class="o">&gt;=</span> <span class="mi">2</span><span class="p">:</span>
</span><span id="flatten_schema-513"><a href="#flatten_schema-513"><span class="linenos">513</span></a> <span class="n">tables</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">flatten_schema</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">depth</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="n">keys</span> <span class="o">+</span> <span class="p">[</span><span class="n">k</span><span class="p">]))</span>
</span><span id="flatten_schema-514"><a href="#flatten_schema-514"><span class="linenos">514</span></a>
</span><span id="flatten_schema-515"><a href="#flatten_schema-515"><span class="linenos">515</span></a> <span class="k">return</span> <span class="n">tables</span>
</span></pre></div>
@ -2250,30 +2262,30 @@ The added table must have the necessary number of qualifiers in its path to matc
</div>
<a class="headerlink" href="#nested_get"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="nested_get-512"><a href="#nested_get-512"><span class="linenos">512</span></a><span class="k">def</span> <span class="nf">nested_get</span><span class="p">(</span>
</span><span id="nested_get-513"><a href="#nested_get-513"><span class="linenos">513</span></a> <span class="n">d</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="o">*</span><span class="n">path</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span> <span class="n">raise_on_missing</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>
</span><span id="nested_get-514"><a href="#nested_get-514"><span class="linenos">514</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">]:</span>
</span><span id="nested_get-515"><a href="#nested_get-515"><span class="linenos">515</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="nested_get-516"><a href="#nested_get-516"><span class="linenos">516</span></a><span class="sd"> Get a value for a nested dictionary.</span>
</span><span id="nested_get-517"><a href="#nested_get-517"><span class="linenos">517</span></a>
</span><span id="nested_get-518"><a href="#nested_get-518"><span class="linenos">518</span></a><span class="sd"> Args:</span>
</span><span id="nested_get-519"><a href="#nested_get-519"><span class="linenos">519</span></a><span class="sd"> d: the dictionary to search.</span>
</span><span id="nested_get-520"><a href="#nested_get-520"><span class="linenos">520</span></a><span class="sd"> *path: tuples of (name, key), where:</span>
</span><span id="nested_get-521"><a href="#nested_get-521"><span class="linenos">521</span></a><span class="sd"> `key` is the key in the dictionary to get.</span>
</span><span id="nested_get-522"><a href="#nested_get-522"><span class="linenos">522</span></a><span class="sd"> `name` is a string to use in the error if `key` isn&#39;t found.</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="nested_get-518"><a href="#nested_get-518"><span class="linenos">518</span></a><span class="k">def</span> <span class="nf">nested_get</span><span class="p">(</span>
</span><span id="nested_get-519"><a href="#nested_get-519"><span class="linenos">519</span></a> <span class="n">d</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="o">*</span><span class="n">path</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span> <span class="n">raise_on_missing</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>
</span><span id="nested_get-520"><a href="#nested_get-520"><span class="linenos">520</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">]:</span>
</span><span id="nested_get-521"><a href="#nested_get-521"><span class="linenos">521</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="nested_get-522"><a href="#nested_get-522"><span class="linenos">522</span></a><span class="sd"> Get a value for a nested dictionary.</span>
</span><span id="nested_get-523"><a href="#nested_get-523"><span class="linenos">523</span></a>
</span><span id="nested_get-524"><a href="#nested_get-524"><span class="linenos">524</span></a><span class="sd"> Returns:</span>
</span><span id="nested_get-525"><a href="#nested_get-525"><span class="linenos">525</span></a><span class="sd"> The value or None if it doesn&#39;t exist.</span>
</span><span id="nested_get-526"><a href="#nested_get-526"><span class="linenos">526</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="nested_get-527"><a href="#nested_get-527"><span class="linenos">527</span></a> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">path</span><span class="p">:</span>
</span><span id="nested_get-528"><a href="#nested_get-528"><span class="linenos">528</span></a> <span class="n">d</span> <span class="o">=</span> <span class="n">d</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="c1"># type: ignore</span>
</span><span id="nested_get-529"><a href="#nested_get-529"><span class="linenos">529</span></a> <span class="k">if</span> <span class="n">d</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span><span id="nested_get-530"><a href="#nested_get-530"><span class="linenos">530</span></a> <span class="k">if</span> <span class="n">raise_on_missing</span><span class="p">:</span>
</span><span id="nested_get-531"><a href="#nested_get-531"><span class="linenos">531</span></a> <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;table&quot;</span> <span class="k">if</span> <span class="n">name</span> <span class="o">==</span> <span class="s2">&quot;this&quot;</span> <span class="k">else</span> <span class="n">name</span>
</span><span id="nested_get-532"><a href="#nested_get-532"><span class="linenos">532</span></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unknown </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="nested_get-533"><a href="#nested_get-533"><span class="linenos">533</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="nested_get-534"><a href="#nested_get-534"><span class="linenos">534</span></a>
</span><span id="nested_get-535"><a href="#nested_get-535"><span class="linenos">535</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="nested_get-524"><a href="#nested_get-524"><span class="linenos">524</span></a><span class="sd"> Args:</span>
</span><span id="nested_get-525"><a href="#nested_get-525"><span class="linenos">525</span></a><span class="sd"> d: the dictionary to search.</span>
</span><span id="nested_get-526"><a href="#nested_get-526"><span class="linenos">526</span></a><span class="sd"> *path: tuples of (name, key), where:</span>
</span><span id="nested_get-527"><a href="#nested_get-527"><span class="linenos">527</span></a><span class="sd"> `key` is the key in the dictionary to get.</span>
</span><span id="nested_get-528"><a href="#nested_get-528"><span class="linenos">528</span></a><span class="sd"> `name` is a string to use in the error if `key` isn&#39;t found.</span>
</span><span id="nested_get-529"><a href="#nested_get-529"><span class="linenos">529</span></a>
</span><span id="nested_get-530"><a href="#nested_get-530"><span class="linenos">530</span></a><span class="sd"> Returns:</span>
</span><span id="nested_get-531"><a href="#nested_get-531"><span class="linenos">531</span></a><span class="sd"> The value or None if it doesn&#39;t exist.</span>
</span><span id="nested_get-532"><a href="#nested_get-532"><span class="linenos">532</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="nested_get-533"><a href="#nested_get-533"><span class="linenos">533</span></a> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">path</span><span class="p">:</span>
</span><span id="nested_get-534"><a href="#nested_get-534"><span class="linenos">534</span></a> <span class="n">d</span> <span class="o">=</span> <span class="n">d</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="c1"># type: ignore</span>
</span><span id="nested_get-535"><a href="#nested_get-535"><span class="linenos">535</span></a> <span class="k">if</span> <span class="n">d</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span><span id="nested_get-536"><a href="#nested_get-536"><span class="linenos">536</span></a> <span class="k">if</span> <span class="n">raise_on_missing</span><span class="p">:</span>
</span><span id="nested_get-537"><a href="#nested_get-537"><span class="linenos">537</span></a> <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;table&quot;</span> <span class="k">if</span> <span class="n">name</span> <span class="o">==</span> <span class="s2">&quot;this&quot;</span> <span class="k">else</span> <span class="n">name</span>
</span><span id="nested_get-538"><a href="#nested_get-538"><span class="linenos">538</span></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unknown </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</span><span id="nested_get-539"><a href="#nested_get-539"><span class="linenos">539</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="nested_get-540"><a href="#nested_get-540"><span class="linenos">540</span></a>
</span><span id="nested_get-541"><a href="#nested_get-541"><span class="linenos">541</span></a> <span class="k">return</span> <span class="n">d</span>
</span></pre></div>
@ -2308,41 +2320,41 @@ The added table must have the necessary number of qualifiers in its path to matc
</div>
<a class="headerlink" href="#nested_set"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="nested_set-538"><a href="#nested_set-538"><span class="linenos">538</span></a><span class="k">def</span> <span class="nf">nested_set</span><span class="p">(</span><span class="n">d</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">keys</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Sequence</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">value</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">:</span>
</span><span id="nested_set-539"><a href="#nested_set-539"><span class="linenos">539</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="nested_set-540"><a href="#nested_set-540"><span class="linenos">540</span></a><span class="sd"> In-place set a value for a nested dictionary</span>
</span><span id="nested_set-541"><a href="#nested_set-541"><span class="linenos">541</span></a>
</span><span id="nested_set-542"><a href="#nested_set-542"><span class="linenos">542</span></a><span class="sd"> Example:</span>
</span><span id="nested_set-543"><a href="#nested_set-543"><span class="linenos">543</span></a><span class="sd"> &gt;&gt;&gt; nested_set({}, [&quot;top_key&quot;, &quot;second_key&quot;], &quot;value&quot;)</span>
</span><span id="nested_set-544"><a href="#nested_set-544"><span class="linenos">544</span></a><span class="sd"> {&#39;top_key&#39;: {&#39;second_key&#39;: &#39;value&#39;}}</span>
</span><span id="nested_set-545"><a href="#nested_set-545"><span class="linenos">545</span></a>
</span><span id="nested_set-546"><a href="#nested_set-546"><span class="linenos">546</span></a><span class="sd"> &gt;&gt;&gt; nested_set({&quot;top_key&quot;: {&quot;third_key&quot;: &quot;third_value&quot;}}, [&quot;top_key&quot;, &quot;second_key&quot;], &quot;value&quot;)</span>
</span><span id="nested_set-547"><a href="#nested_set-547"><span class="linenos">547</span></a><span class="sd"> {&#39;top_key&#39;: {&#39;third_key&#39;: &#39;third_value&#39;, &#39;second_key&#39;: &#39;value&#39;}}</span>
</span><span id="nested_set-548"><a href="#nested_set-548"><span class="linenos">548</span></a>
</span><span id="nested_set-549"><a href="#nested_set-549"><span class="linenos">549</span></a><span class="sd"> Args:</span>
</span><span id="nested_set-550"><a href="#nested_set-550"><span class="linenos">550</span></a><span class="sd"> d: dictionary to update.</span>
</span><span id="nested_set-551"><a href="#nested_set-551"><span class="linenos">551</span></a><span class="sd"> keys: the keys that makeup the path to `value`.</span>
</span><span id="nested_set-552"><a href="#nested_set-552"><span class="linenos">552</span></a><span class="sd"> value: the value to set in the dictionary for the given key path.</span>
</span><span id="nested_set-553"><a href="#nested_set-553"><span class="linenos">553</span></a>
</span><span id="nested_set-554"><a href="#nested_set-554"><span class="linenos">554</span></a><span class="sd"> Returns:</span>
</span><span id="nested_set-555"><a href="#nested_set-555"><span class="linenos">555</span></a><span class="sd"> The (possibly) updated dictionary.</span>
</span><span id="nested_set-556"><a href="#nested_set-556"><span class="linenos">556</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="nested_set-557"><a href="#nested_set-557"><span class="linenos">557</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="nested_set-558"><a href="#nested_set-558"><span class="linenos">558</span></a> <span class="k">return</span> <span class="n">d</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="nested_set-544"><a href="#nested_set-544"><span class="linenos">544</span></a><span class="k">def</span> <span class="nf">nested_set</span><span class="p">(</span><span class="n">d</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">,</span> <span class="n">keys</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Sequence</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">value</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">:</span>
</span><span id="nested_set-545"><a href="#nested_set-545"><span class="linenos">545</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="nested_set-546"><a href="#nested_set-546"><span class="linenos">546</span></a><span class="sd"> In-place set a value for a nested dictionary</span>
</span><span id="nested_set-547"><a href="#nested_set-547"><span class="linenos">547</span></a>
</span><span id="nested_set-548"><a href="#nested_set-548"><span class="linenos">548</span></a><span class="sd"> Example:</span>
</span><span id="nested_set-549"><a href="#nested_set-549"><span class="linenos">549</span></a><span class="sd"> &gt;&gt;&gt; nested_set({}, [&quot;top_key&quot;, &quot;second_key&quot;], &quot;value&quot;)</span>
</span><span id="nested_set-550"><a href="#nested_set-550"><span class="linenos">550</span></a><span class="sd"> {&#39;top_key&#39;: {&#39;second_key&#39;: &#39;value&#39;}}</span>
</span><span id="nested_set-551"><a href="#nested_set-551"><span class="linenos">551</span></a>
</span><span id="nested_set-552"><a href="#nested_set-552"><span class="linenos">552</span></a><span class="sd"> &gt;&gt;&gt; nested_set({&quot;top_key&quot;: {&quot;third_key&quot;: &quot;third_value&quot;}}, [&quot;top_key&quot;, &quot;second_key&quot;], &quot;value&quot;)</span>
</span><span id="nested_set-553"><a href="#nested_set-553"><span class="linenos">553</span></a><span class="sd"> {&#39;top_key&#39;: {&#39;third_key&#39;: &#39;third_value&#39;, &#39;second_key&#39;: &#39;value&#39;}}</span>
</span><span id="nested_set-554"><a href="#nested_set-554"><span class="linenos">554</span></a>
</span><span id="nested_set-555"><a href="#nested_set-555"><span class="linenos">555</span></a><span class="sd"> Args:</span>
</span><span id="nested_set-556"><a href="#nested_set-556"><span class="linenos">556</span></a><span class="sd"> d: dictionary to update.</span>
</span><span id="nested_set-557"><a href="#nested_set-557"><span class="linenos">557</span></a><span class="sd"> keys: the keys that makeup the path to `value`.</span>
</span><span id="nested_set-558"><a href="#nested_set-558"><span class="linenos">558</span></a><span class="sd"> value: the value to set in the dictionary for the given key path.</span>
</span><span id="nested_set-559"><a href="#nested_set-559"><span class="linenos">559</span></a>
</span><span id="nested_set-560"><a href="#nested_set-560"><span class="linenos">560</span></a> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="nested_set-561"><a href="#nested_set-561"><span class="linenos">561</span></a> <span class="n">d</span><span class="p">[</span><span class="n">keys</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value</span>
</span><span id="nested_set-562"><a href="#nested_set-562"><span class="linenos">562</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="nested_set-563"><a href="#nested_set-563"><span class="linenos">563</span></a>
</span><span id="nested_set-564"><a href="#nested_set-564"><span class="linenos">564</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">d</span>
</span><span id="nested_set-565"><a href="#nested_set-565"><span class="linenos">565</span></a> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
</span><span id="nested_set-566"><a href="#nested_set-566"><span class="linenos">566</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">subd</span><span class="p">:</span>
</span><span id="nested_set-567"><a href="#nested_set-567"><span class="linenos">567</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">subd</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="p">{})</span>
</span><span id="nested_set-568"><a href="#nested_set-568"><span class="linenos">568</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="nested_set-569"><a href="#nested_set-569"><span class="linenos">569</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">subd</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
</span><span id="nested_set-570"><a href="#nested_set-570"><span class="linenos">570</span></a>
</span><span id="nested_set-571"><a href="#nested_set-571"><span class="linenos">571</span></a> <span class="n">subd</span><span class="p">[</span><span class="n">keys</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value</span>
</span><span id="nested_set-572"><a href="#nested_set-572"><span class="linenos">572</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="nested_set-560"><a href="#nested_set-560"><span class="linenos">560</span></a><span class="sd"> Returns:</span>
</span><span id="nested_set-561"><a href="#nested_set-561"><span class="linenos">561</span></a><span class="sd"> The (possibly) updated dictionary.</span>
</span><span id="nested_set-562"><a href="#nested_set-562"><span class="linenos">562</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="nested_set-563"><a href="#nested_set-563"><span class="linenos">563</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">keys</span><span class="p">:</span>
</span><span id="nested_set-564"><a href="#nested_set-564"><span class="linenos">564</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="nested_set-565"><a href="#nested_set-565"><span class="linenos">565</span></a>
</span><span id="nested_set-566"><a href="#nested_set-566"><span class="linenos">566</span></a> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span><span id="nested_set-567"><a href="#nested_set-567"><span class="linenos">567</span></a> <span class="n">d</span><span class="p">[</span><span class="n">keys</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value</span>
</span><span id="nested_set-568"><a href="#nested_set-568"><span class="linenos">568</span></a> <span class="k">return</span> <span class="n">d</span>
</span><span id="nested_set-569"><a href="#nested_set-569"><span class="linenos">569</span></a>
</span><span id="nested_set-570"><a href="#nested_set-570"><span class="linenos">570</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">d</span>
</span><span id="nested_set-571"><a href="#nested_set-571"><span class="linenos">571</span></a> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
</span><span id="nested_set-572"><a href="#nested_set-572"><span class="linenos">572</span></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">subd</span><span class="p">:</span>
</span><span id="nested_set-573"><a href="#nested_set-573"><span class="linenos">573</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">subd</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="p">{})</span>
</span><span id="nested_set-574"><a href="#nested_set-574"><span class="linenos">574</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="nested_set-575"><a href="#nested_set-575"><span class="linenos">575</span></a> <span class="n">subd</span> <span class="o">=</span> <span class="n">subd</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
</span><span id="nested_set-576"><a href="#nested_set-576"><span class="linenos">576</span></a>
</span><span id="nested_set-577"><a href="#nested_set-577"><span class="linenos">577</span></a> <span class="n">subd</span><span class="p">[</span><span class="n">keys</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value</span>
</span><span id="nested_set-578"><a href="#nested_set-578"><span class="linenos">578</span></a> <span class="k">return</span> <span class="n">d</span>
</span></pre></div>

2
docs/sqlglot/time.html
View file

File diff suppressed because one or more lines are too long

4588
docs/sqlglot/tokens.html
View file

File diff suppressed because it is too large Load diff

2090
docs/sqlglot/transforms.html
View file

File diff suppressed because it is too large Load diff

13
setup.cfg
View file

@ -13,16 +13,3 @@ ignore_errors = True
[mypy-tests.dataframe.*]
ignore_errors = False
[autoflake]
in-place = True
expand-star-imports = True
remove-all-unused-imports = True
ignore-init-module-imports = True
remove-duplicate-keys = True
remove-unused-variables = True
quiet = True
[isort]
profile=black
known_first_party=sqlglot

2
setup.py
View file

@ -37,7 +37,7 @@ setup(
"python-dateutil",
"pdoc",
"pre-commit",
"ruff==0.3.2",
"ruff==0.4.3",
"types-python-dateutil",
"typing_extensions",
"maturin>=1.4,<2.0",

5
sqlglot/dialects/athena.py
View file

@ -13,6 +13,8 @@ class Athena(Trino):
}
class Generator(Trino.Generator):
WITH_PROPERTIES_PREFIX = "TBLPROPERTIES"
PROPERTIES_LOCATION = {
**Trino.Generator.PROPERTIES_LOCATION,
exp.LocationProperty: exp.Properties.Location.POST_SCHEMA,
@ -32,6 +34,3 @@ class Athena(Trino):
return (
f"{self.property_name(expression, string_key=True)}={self.sql(expression, 'value')}"
)
def with_properties(self, properties: exp.Properties) -> str:
return self.properties(properties, prefix=self.seg("TBLPROPERTIES"))

21
sqlglot/dialects/bigquery.py
View file

@ -156,7 +156,7 @@ def _build_date(args: t.List) -> exp.Date | exp.DateFromParts:
def _build_to_hex(args: t.List) -> exp.Hex | exp.MD5:
# TO_HEX(MD5(..)) is common in BigQuery, so it's parsed into MD5 to simplify its transpilation
arg = seq_get(args, 0)
return exp.MD5(this=arg.this) if isinstance(arg, exp.MD5Digest) else exp.Hex(this=arg)
return exp.MD5(this=arg.this) if isinstance(arg, exp.MD5Digest) else exp.LowerHex(this=arg)
def _array_contains_sql(self: BigQuery.Generator, expression: exp.ArrayContains) -> str:
@ -212,6 +212,7 @@ class BigQuery(Dialect):
SUPPORTS_USER_DEFINED_TYPES = False
SUPPORTS_SEMI_ANTI_JOIN = False
LOG_BASE_FIRST = False
HEX_LOWERCASE = True
# https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical#case_sensitivity
NORMALIZATION_STRATEGY = NormalizationStrategy.CASE_INSENSITIVE
@ -568,6 +569,8 @@ class BigQuery(Dialect):
CAN_IMPLEMENT_ARRAY_ANY = True
SUPPORTS_TO_NUMBER = False
NAMED_PLACEHOLDER_TOKEN = "@"
HEX_FUNC = "TO_HEX"
WITH_PROPERTIES_PREFIX = "OPTIONS"
TRANSFORMS = {
**generator.Generator.TRANSFORMS,
@ -603,13 +606,12 @@ class BigQuery(Dialect):
),
exp.GenerateSeries: rename_func("GENERATE_ARRAY"),
exp.GroupConcat: rename_func("STRING_AGG"),
exp.Hex: rename_func("TO_HEX"),
exp.Hex: lambda self, e: self.func("UPPER", self.func("TO_HEX", self.sql(e, "this"))),
exp.If: if_sql(false_value="NULL"),
exp.ILike: no_ilike_sql,
exp.IntDiv: rename_func("DIV"),
exp.JSONFormat: rename_func("TO_JSON_STRING"),
exp.Max: max_or_greatest,
exp.Mod: rename_func("MOD"),
exp.MD5: lambda self, e: self.func("TO_HEX", self.func("MD5", e.this)),
exp.MD5Digest: rename_func("MD5"),
exp.Min: min_or_least,
@ -634,6 +636,7 @@ class BigQuery(Dialect):
transforms.eliminate_semi_and_anti_joins,
]
),
exp.SHA: rename_func("SHA1"),
exp.SHA2: lambda self, e: self.func(
"SHA256" if e.text("length") == "256" else "SHA512", e.this
),
@ -801,6 +804,15 @@ class BigQuery(Dialect):
"within",
}
def mod_sql(self, expression: exp.Mod) -> str:
this = expression.this
expr = expression.expression
return self.func(
"MOD",
this.unnest() if isinstance(this, exp.Paren) else this,
expr.unnest() if isinstance(expr, exp.Paren) else expr,
)
def column_parts(self, expression: exp.Column) -> str:
if expression.meta.get("quoted_column"):
# If a column reference is of the form `dataset.table`.name, we need
@ -896,9 +908,6 @@ class BigQuery(Dialect):
self.unsupported("INTERSECT without DISTINCT is not supported in BigQuery")
return f"INTERSECT{' DISTINCT' if expression.args.get('distinct') else ' ALL'}"
def with_properties(self, properties: exp.Properties) -> str:
return self.properties(properties, prefix=self.seg("OPTIONS"))
def version_sql(self, expression: exp.Version) -> str:
if expression.name == "TIMESTAMP":
expression.set("this", "SYSTEM_TIME")

92
sqlglot/dialects/clickhouse.py
View file

@ -15,6 +15,7 @@ from sqlglot.dialects.dialect import (
build_json_extract_path,
rename_func,
var_map_sql,
timestamptrunc_sql,
)
from sqlglot.helper import is_int, seq_get
from sqlglot.tokens import Token, TokenType
@ -30,6 +31,27 @@ def _build_date_format(args: t.List) -> exp.TimeToStr:
return expr
def _unix_to_time_sql(self: ClickHouse.Generator, expression: exp.UnixToTime) -> str:
scale = expression.args.get("scale")
timestamp = expression.this
if scale in (None, exp.UnixToTime.SECONDS):
return self.func("fromUnixTimestamp", exp.cast(timestamp, exp.DataType.Type.BIGINT))
if scale == exp.UnixToTime.MILLIS:
return self.func("fromUnixTimestamp64Milli", exp.cast(timestamp, exp.DataType.Type.BIGINT))
if scale == exp.UnixToTime.MICROS:
return self.func("fromUnixTimestamp64Micro", exp.cast(timestamp, exp.DataType.Type.BIGINT))
if scale == exp.UnixToTime.NANOS:
return self.func("fromUnixTimestamp64Nano", exp.cast(timestamp, exp.DataType.Type.BIGINT))
return self.func(
"fromUnixTimestamp",
exp.cast(
exp.Div(this=timestamp, expression=exp.func("POW", 10, scale)), exp.DataType.Type.BIGINT
),
)
def _lower_func(sql: str) -> str:
index = sql.index("(")
return sql[:index].lower() + sql[index:]
@ -146,6 +168,9 @@ class ClickHouse(Dialect):
"TUPLE": exp.Struct.from_arg_list,
"UNIQ": exp.ApproxDistinct.from_arg_list,
"XOR": lambda args: exp.Xor(expressions=args),
"MD5": exp.MD5Digest.from_arg_list,
"SHA256": lambda args: exp.SHA2(this=seq_get(args, 0), length=exp.Literal.number(256)),
"SHA512": lambda args: exp.SHA2(this=seq_get(args, 0), length=exp.Literal.number(512)),
}
AGG_FUNCTIONS = {
@ -353,6 +378,11 @@ class ClickHouse(Dialect):
"CODEC": lambda self: self._parse_compress(),
}
ALTER_PARSERS = {
**parser.Parser.ALTER_PARSERS,
"REPLACE": lambda self: self._parse_alter_table_replace(),
}
SCHEMA_UNNAMED_CONSTRAINTS = {
*parser.Parser.SCHEMA_UNNAMED_CONSTRAINTS,
"INDEX",
@ -578,6 +608,44 @@ class ClickHouse(Dialect):
granularity=granularity,
)
def _parse_partition(self) -> t.Optional[exp.Partition]:
# https://clickhouse.com/docs/en/sql-reference/statements/alter/partition#how-to-set-partition-expression
if not self._match(TokenType.PARTITION):
return None
if self._match_text_seq("ID"):
# Corresponds to the PARTITION ID <string_value> syntax
expressions: t.List[exp.Expression] = [
self.expression(exp.PartitionId, this=self._parse_string())
]
else:
expressions = self._parse_expressions()
return self.expression(exp.Partition, expressions=expressions)
def _parse_alter_table_replace(self) -> t.Optional[exp.Expression]:
partition = self._parse_partition()
if not partition or not self._match(TokenType.FROM):
return None
return self.expression(
exp.ReplacePartition, expression=partition, source=self._parse_table_parts()
)
def _parse_projection_def(self) -> t.Optional[exp.ProjectionDef]:
if not self._match_text_seq("PROJECTION"):
return None
return self.expression(
exp.ProjectionDef,
this=self._parse_id_var(),
expression=self._parse_wrapped(self._parse_statement),
)
def _parse_constraint(self) -> t.Optional[exp.Expression]:
return super()._parse_constraint() or self._parse_projection_def()
class Generator(generator.Generator):
QUERY_HINTS = False
STRUCT_DELIMITER = ("(", ")")
@ -687,6 +755,16 @@ class ClickHouse(Dialect):
),
exp.VarMap: lambda self, e: _lower_func(var_map_sql(self, e)),
exp.Xor: lambda self, e: self.func("xor", e.this, e.expression, *e.expressions),
exp.MD5Digest: rename_func("MD5"),
exp.MD5: lambda self, e: self.func("LOWER", self.func("HEX", self.func("MD5", e.this))),
exp.SHA: rename_func("SHA1"),
exp.SHA2: lambda self, e: self.func(
"SHA256" if e.text("length") == "256" else "SHA512", e.this
),
exp.UnixToTime: _unix_to_time_sql,
exp.TimestampTrunc: timestamptrunc_sql(zone=True),
exp.Variance: rename_func("varSamp"),
exp.Stddev: rename_func("stddevSamp"),
}
PROPERTIES_LOCATION = {
@ -828,3 +906,17 @@ class ClickHouse(Dialect):
granularity = f" GRANULARITY {granularity}" if granularity else ""
return f"INDEX{this}{expr}{index_type}{granularity}"
def partition_sql(self, expression: exp.Partition) -> str:
return f"PARTITION {self.expressions(expression, flat=True)}"
def partitionid_sql(self, expression: exp.PartitionId) -> str:
return f"ID {self.sql(expression.this)}"
def replacepartition_sql(self, expression: exp.ReplacePartition) -> str:
return (
f"REPLACE {self.sql(expression.expression)} FROM {self.sql(expression, 'source')}"
)
def projectiondef_sql(self, expression: exp.ProjectionDef) -> str:
return f"PROJECTION {self.sql(expression.this)} {self.wrap(expression.expression)}"

2
sqlglot/dialects/databricks.py
View file

@ -57,7 +57,7 @@ class Databricks(Spark):
),
exp.DatetimeDiff: _timestamp_diff,
exp.TimestampDiff: _timestamp_diff,
exp.DatetimeTrunc: timestamptrunc_sql,
exp.DatetimeTrunc: timestamptrunc_sql(),
exp.JSONExtract: lambda self, e: self.binary(e, ":"),
exp.Select: transforms.preprocess(
[

28
sqlglot/dialects/dialect.py
View file

@ -248,6 +248,9 @@ class Dialect(metaclass=_Dialect):
CONCAT_COALESCE = False
"""A `NULL` arg in `CONCAT` yields `NULL` by default, but in some dialects it yields an empty string."""
HEX_LOWERCASE = False
"""Whether the `HEX` function returns a lowercase hexadecimal string."""
DATE_FORMAT = "'%Y-%m-%d'"
DATEINT_FORMAT = "'%Y%m%d'"
TIME_FORMAT = "'%Y-%m-%d %H:%M:%S'"
@ -769,8 +772,14 @@ def date_add_interval_sql(
return func
def timestamptrunc_sql(self: Generator, expression: exp.TimestampTrunc) -> str:
return self.func("DATE_TRUNC", unit_to_str(expression), expression.this)
def timestamptrunc_sql(zone: bool = False) -> t.Callable[[Generator, exp.TimestampTrunc], str]:
def _timestamptrunc_sql(self: Generator, expression: exp.TimestampTrunc) -> str:
args = [unit_to_str(expression), expression.this]
if zone:
args.append(expression.args.get("zone"))
return self.func("DATE_TRUNC", *args)
return _timestamptrunc_sql
def no_timestamp_sql(self: Generator, expression: exp.Timestamp) -> str:
@ -1141,10 +1150,23 @@ def filter_array_using_unnest(self: Generator, expression: exp.ArrayFilter) -> s
return self.sql(exp.Array(expressions=[filtered]))
def to_number_with_nls_param(self, expression: exp.ToNumber) -> str:
def to_number_with_nls_param(self: Generator, expression: exp.ToNumber) -> str:
return self.func(
"TO_NUMBER",
expression.this,
expression.args.get("format"),
expression.args.get("nlsparam"),
)
def build_default_decimal_type(
precision: t.Optional[int] = None, scale: t.Optional[int] = None
) -> t.Callable[[exp.DataType], exp.DataType]:
def _builder(dtype: exp.DataType) -> exp.DataType:
if dtype.expressions or precision is None:
return dtype
params = f"{precision}{f', {scale}' if scale is not None else ''}"
return exp.DataType.build(f"DECIMAL({params})")
return _builder

62
sqlglot/dialects/duckdb.py
View file

@ -5,12 +5,14 @@ import typing as t
from sqlglot import exp, generator, parser, tokens, transforms
from sqlglot.dialects.dialect import (
Dialect,
JSON_EXTRACT_TYPE,
NormalizationStrategy,
approx_count_distinct_sql,
arg_max_or_min_no_count,
arrow_json_extract_sql,
binary_from_function,
bool_xor_sql,
build_default_decimal_type,
date_trunc_to_time,
datestrtodate_sql,
encode_decode_sql,
@ -155,6 +157,13 @@ def _unix_to_time_sql(self: DuckDB.Generator, expression: exp.UnixToTime) -> str
return self.func("TO_TIMESTAMP", exp.Div(this=timestamp, expression=exp.func("POW", 10, scale)))
def _arrow_json_extract_sql(self: DuckDB.Generator, expression: JSON_EXTRACT_TYPE) -> str:
arrow_sql = arrow_json_extract_sql(self, expression)
if not expression.same_parent and isinstance(expression.parent, exp.Binary):
arrow_sql = self.wrap(arrow_sql)
return arrow_sql
class DuckDB(Dialect):
NULL_ORDERING = "nulls_are_last"
SUPPORTS_USER_DEFINED_TYPES = False
@ -304,6 +313,22 @@ class DuckDB(Dialect):
),
}
TYPE_CONVERTER = {
# https://duckdb.org/docs/sql/data_types/numeric
exp.DataType.Type.DECIMAL: build_default_decimal_type(precision=18, scale=3),
}
def _parse_table_sample(self, as_modifier: bool = False) -> t.Optional[exp.TableSample]:
# https://duckdb.org/docs/sql/samples.html
sample = super()._parse_table_sample(as_modifier=as_modifier)
if sample and not sample.args.get("method"):
if sample.args.get("size"):
sample.set("method", exp.var("RESERVOIR"))
else:
sample.set("method", exp.var("SYSTEM"))
return sample
def _parse_bracket(
self, this: t.Optional[exp.Expression] = None
) -> t.Optional[exp.Expression]:
@ -320,24 +345,6 @@ class DuckDB(Dialect):
args = self._parse_wrapped_csv(self._parse_conjunction)
return self.expression(exp.Map, keys=seq_get(args, 0), values=seq_get(args, 1))
def _parse_types(
self, check_func: bool = False, schema: bool = False, allow_identifiers: bool = True
) -> t.Optional[exp.Expression]:
this = super()._parse_types(
check_func=check_func, schema=schema, allow_identifiers=allow_identifiers
)
# DuckDB treats NUMERIC and DECIMAL without precision as DECIMAL(18, 3)
# See: https://duckdb.org/docs/sql/data_types/numeric
if (
isinstance(this, exp.DataType)
and this.is_type("numeric", "decimal")
and not this.expressions
):
return exp.DataType.build("DECIMAL(18, 3)")
return this
def _parse_struct_types(self, type_required: bool = False) -> t.Optional[exp.Expression]:
return self._parse_field_def()
@ -368,6 +375,7 @@ class DuckDB(Dialect):
CAN_IMPLEMENT_ARRAY_ANY = True
SUPPORTS_TO_NUMBER = False
COPY_HAS_INTO_KEYWORD = False
STAR_EXCEPT = "EXCLUDE"
TRANSFORMS = {
**generator.Generator.TRANSFORMS,
@ -406,11 +414,12 @@ class DuckDB(Dialect):
exp.IntDiv: lambda self, e: self.binary(e, "//"),
exp.IsInf: rename_func("ISINF"),
exp.IsNan: rename_func("ISNAN"),
exp.JSONExtract: arrow_json_extract_sql,
exp.JSONExtractScalar: arrow_json_extract_sql,
exp.JSONExtract: _arrow_json_extract_sql,
exp.JSONExtractScalar: _arrow_json_extract_sql,
exp.JSONFormat: _json_format_sql,
exp.LogicalOr: rename_func("BOOL_OR"),
exp.LogicalAnd: rename_func("BOOL_AND"),
exp.MD5Digest: lambda self, e: self.func("UNHEX", self.func("MD5", e.this)),
exp.MonthsBetween: lambda self, e: self.func(
"DATEDIFF",
"'month'",
@ -449,7 +458,7 @@ class DuckDB(Dialect):
exp.TimestampDiff: lambda self, e: self.func(
"DATE_DIFF", exp.Literal.string(e.unit), e.expression, e.this
),
exp.TimestampTrunc: timestamptrunc_sql,
exp.TimestampTrunc: timestamptrunc_sql(),
exp.TimeStrToDate: lambda self, e: self.sql(exp.cast(e.this, exp.DataType.Type.DATE)),
exp.TimeStrToTime: timestrtotime_sql,
exp.TimeStrToUnix: lambda self, e: self.func(
@ -499,8 +508,6 @@ class DuckDB(Dialect):
exp.DataType.Type.TIMESTAMP_NS: "TIMESTAMP_NS",
}
STAR_MAPPING = {**generator.Generator.STAR_MAPPING, "except": "EXCLUDE"}
UNWRAPPED_INTERVAL_VALUES = (exp.Literal, exp.Paren)
# DuckDB doesn't generally support CREATE TABLE .. properties
@ -550,6 +557,15 @@ class DuckDB(Dialect):
# This sample clause only applies to a single source, not the entire resulting relation
tablesample_keyword = "TABLESAMPLE"
if expression.args.get("size"):
method = expression.args.get("method")
if method and method.name.upper() != "RESERVOIR":
self.unsupported(
f"Sampling method {method} is not supported with a discrete sample count, "
"defaulting to reservoir sampling"
)
expression.set("method", exp.var("RESERVOIR"))
return super().tablesample_sql(
expression, sep=sep, tablesample_keyword=tablesample_keyword
)

41
sqlglot/dialects/hive.py
View file

@ -254,7 +254,7 @@ class Hive(Dialect):
"REFRESH": TokenType.REFRESH,
"TIMESTAMP AS OF": TokenType.TIMESTAMP_SNAPSHOT,
"VERSION AS OF": TokenType.VERSION_SNAPSHOT,
"WITH SERDEPROPERTIES": TokenType.SERDE_PROPERTIES,
"SERDEPROPERTIES": TokenType.SERDE_PROPERTIES,
}
NUMERIC_LITERALS = {
@ -332,7 +332,7 @@ class Hive(Dialect):
PROPERTY_PARSERS = {
**parser.Parser.PROPERTY_PARSERS,
"WITH SERDEPROPERTIES": lambda self: exp.SerdeProperties(
"SERDEPROPERTIES": lambda self: exp.SerdeProperties(
expressions=self._parse_wrapped_csv(self._parse_property)
),
}
@ -422,6 +422,15 @@ class Hive(Dialect):
super()._parse_order(skip_order_token=self._match(TokenType.SORT_BY)),
)
def _parse_parameter(self) -> exp.Parameter:
self._match(TokenType.L_BRACE)
this = self._parse_identifier() or self._parse_primary_or_var()
expression = self._match(TokenType.COLON) and (
self._parse_identifier() or self._parse_primary_or_var()
)
self._match(TokenType.R_BRACE)
return self.expression(exp.Parameter, this=this, expression=expression)
class Generator(generator.Generator):
LIMIT_FETCH = "LIMIT"
TABLESAMPLE_WITH_METHOD = False
@ -434,6 +443,7 @@ class Hive(Dialect):
LAST_DAY_SUPPORTS_DATE_PART = False
JSON_PATH_SINGLE_QUOTE_ESCAPE = True
SUPPORTS_TO_NUMBER = False
WITH_PROPERTIES_PREFIX = "TBLPROPERTIES"
EXPRESSIONS_WITHOUT_NESTED_CTES = {
exp.Insert,
@ -453,11 +463,12 @@ class Hive(Dialect):
**generator.Generator.TYPE_MAPPING,
exp.DataType.Type.BIT: "BOOLEAN",
exp.DataType.Type.DATETIME: "TIMESTAMP",
exp.DataType.Type.ROWVERSION: "BINARY",
exp.DataType.Type.TEXT: "STRING",
exp.DataType.Type.TIME: "TIMESTAMP",
exp.DataType.Type.TIMESTAMPTZ: "TIMESTAMP",
exp.DataType.Type.UTINYINT: "SMALLINT",
exp.DataType.Type.VARBINARY: "BINARY",
exp.DataType.Type.ROWVERSION: "BINARY",
}
TRANSFORMS = {
@ -552,7 +563,6 @@ class Hive(Dialect):
exp.UnixToTime: _unix_to_time_sql,
exp.UnixToTimeStr: rename_func("FROM_UNIXTIME"),
exp.PartitionedByProperty: lambda self, e: f"PARTITIONED BY {self.sql(e, 'this')}",
exp.SerdeProperties: lambda self, e: self.properties(e, prefix="WITH SERDEPROPERTIES"),
exp.NumberToStr: rename_func("FORMAT_NUMBER"),
exp.National: lambda self, e: self.national_sql(e, prefix=""),
exp.ClusteredColumnConstraint: lambda self,
@ -618,9 +628,6 @@ class Hive(Dialect):
expression.this.this if isinstance(expression.this, exp.Order) else expression.this,
)
def with_properties(self, properties: exp.Properties) -> str:
return self.properties(properties, prefix=self.seg("TBLPROPERTIES"))
def datatype_sql(self, expression: exp.DataType) -> str:
if expression.this in self.PARAMETERIZABLE_TEXT_TYPES and (
not expression.expressions or expression.expressions[0].name == "MAX"
@ -655,3 +662,23 @@ class Hive(Dialect):
values.append(e)
return self.func("STRUCT", *values)
def alterset_sql(self, expression: exp.AlterSet) -> str:
exprs = self.expressions(expression, flat=True)
exprs = f" {exprs}" if exprs else ""
location = self.sql(expression, "location")
location = f" LOCATION {location}" if location else ""
file_format = self.expressions(expression, key="file_format", flat=True, sep=" ")
file_format = f" FILEFORMAT {file_format}" if file_format else ""
serde = self.sql(expression, "serde")
serde = f" SERDE {serde}" if serde else ""
tags = self.expressions(expression, key="tag", flat=True, sep="")
tags = f" TAGS {tags}" if tags else ""
return f"SET{serde}{exprs}{location}{file_format}{tags}"
def serdeproperties_sql(self, expression: exp.SerdeProperties) -> str:
prefix = "WITH " if expression.args.get("with") else ""
exprs = self.expressions(expression, flat=True)
return f"{prefix}SERDEPROPERTIES ({exprs})"

21
sqlglot/dialects/mysql.py
View file

@ -110,6 +110,20 @@ def _trim_sql(self: MySQL.Generator, expression: exp.Trim) -> str:
return f"TRIM({trim_type}{remove_chars}{from_part}{target})"
def _unix_to_time_sql(self: MySQL.Generator, expression: exp.UnixToTime) -> str:
scale = expression.args.get("scale")
timestamp = expression.this
if scale in (None, exp.UnixToTime.SECONDS):
return self.func("FROM_UNIXTIME", timestamp, self.format_time(expression))
return self.func(
"FROM_UNIXTIME",
exp.Div(this=timestamp, expression=exp.func("POW", 10, scale)),
self.format_time(expression),
)
def date_add_sql(
kind: str,
) -> t.Callable[[generator.Generator, exp.Expression], str]:
@ -251,7 +265,7 @@ class MySQL(Dialect):
"@@": TokenType.SESSION_PARAMETER,
}
COMMANDS = tokens.Tokenizer.COMMANDS - {TokenType.SHOW}
COMMANDS = {*tokens.Tokenizer.COMMANDS, TokenType.REPLACE} - {TokenType.SHOW}
class Parser(parser.Parser):
FUNC_TOKENS = {
@ -723,7 +737,7 @@ class MySQL(Dialect):
exp.TsOrDsAdd: date_add_sql("ADD"),
exp.TsOrDsDiff: lambda self, e: self.func("DATEDIFF", e.this, e.expression),
exp.TsOrDsToDate: _ts_or_ds_to_date_sql,
exp.UnixToTime: lambda self, e: self.func("FROM_UNIXTIME", e.this, self.format_time(e)),
exp.UnixToTime: _unix_to_time_sql,
exp.Week: _remove_ts_or_ds_to_date(),
exp.WeekOfYear: _remove_ts_or_ds_to_date(rename_func("WEEKOFYEAR")),
exp.Year: _remove_ts_or_ds_to_date(),
@ -805,6 +819,9 @@ class MySQL(Dialect):
exp.DataType.Type.TIMESTAMPLTZ,
}
def dpipe_sql(self, expression: exp.DPipe) -> str:
return self.func("CONCAT", *expression.flatten())
def extract_sql(self, expression: exp.Extract) -> str:
unit = expression.name
if unit and unit.lower() == "epoch":

30
sqlglot/dialects/postgres.py
View file

@ -227,12 +227,27 @@ def _build_regexp_replace(args: t.List) -> exp.RegexpReplace:
return exp.RegexpReplace.from_arg_list(args)
def _unix_to_time_sql(self: Postgres.Generator, expression: exp.UnixToTime) -> str:
scale = expression.args.get("scale")
timestamp = expression.this
if scale in (None, exp.UnixToTime.SECONDS):
return self.func("TO_TIMESTAMP", timestamp, self.format_time(expression))
return self.func(
"TO_TIMESTAMP",
exp.Div(this=timestamp, expression=exp.func("POW", 10, scale)),
self.format_time(expression),
)
class Postgres(Dialect):
INDEX_OFFSET = 1
TYPED_DIVISION = True
CONCAT_COALESCE = True
NULL_ORDERING = "nulls_are_large"
TIME_FORMAT = "'YYYY-MM-DD HH24:MI:SS'"
TABLESAMPLE_SIZE_IS_PERCENT = True
TIME_MAPPING = {
"AM": "%p",
@ -528,7 +543,7 @@ class Postgres(Dialect):
exp.Substring: _substring_sql,
exp.TimeFromParts: rename_func("MAKE_TIME"),
exp.TimestampFromParts: rename_func("MAKE_TIMESTAMP"),
exp.TimestampTrunc: timestamptrunc_sql,
exp.TimestampTrunc: timestamptrunc_sql(zone=True),
exp.TimeStrToTime: timestrtotime_sql,
exp.TimeToStr: lambda self, e: self.func("TO_CHAR", e.this, self.format_time(e)),
exp.ToChar: lambda self, e: self.function_fallback_sql(e),
@ -543,6 +558,7 @@ class Postgres(Dialect):
exp.VariancePop: rename_func("VAR_POP"),
exp.Variance: rename_func("VAR_SAMP"),
exp.Xor: bool_xor_sql,
exp.UnixToTime: _unix_to_time_sql,
}
TRANSFORMS.pop(exp.CommentColumnConstraint)
@ -593,3 +609,15 @@ class Postgres(Dialect):
expressions = [f"{self.sql(e)} @@ {this}" for e in expression.expressions]
sql = " OR ".join(expressions)
return f"({sql})" if len(expressions) > 1 else sql
def alterset_sql(self, expression: exp.AlterSet) -> str:
exprs = self.expressions(expression, flat=True)
exprs = f"({exprs})" if exprs else ""
access_method = self.sql(expression, "access_method")
access_method = f"ACCESS METHOD {access_method}" if access_method else ""
tablespace = self.sql(expression, "tablespace")
tablespace = f"TABLESPACE {tablespace}" if tablespace else ""
option = self.sql(expression, "option")
return f"SET {exprs}{access_method}{tablespace}{option}"

39
sqlglot/dialects/presto.py
View file

@ -32,6 +32,7 @@ from sqlglot.dialects.hive import Hive
from sqlglot.dialects.mysql import MySQL
from sqlglot.helper import apply_index_offset, seq_get
from sqlglot.tokens import TokenType
from sqlglot.transforms import unqualify_columns
def _explode_to_unnest_sql(self: Presto.Generator, expression: exp.Lateral) -> str:
@ -276,11 +277,13 @@ class Presto(Dialect):
this=seq_get(args, 0), substr=seq_get(args, 1), instance=seq_get(args, 2)
),
"TO_CHAR": _build_to_char,
"TO_HEX": exp.Hex.from_arg_list,
"TO_UNIXTIME": exp.TimeToUnix.from_arg_list,
"TO_UTF8": lambda args: exp.Encode(
this=seq_get(args, 0), charset=exp.Literal.string("utf-8")
),
"MD5": exp.MD5Digest.from_arg_list,
"SHA256": lambda args: exp.SHA2(this=seq_get(args, 0), length=exp.Literal.number(256)),
"SHA512": lambda args: exp.SHA2(this=seq_get(args, 0), length=exp.Literal.number(512)),
}
FUNCTION_PARSERS = parser.Parser.FUNCTION_PARSERS.copy()
@ -300,6 +303,7 @@ class Presto(Dialect):
LIKE_PROPERTY_INSIDE_SCHEMA = True
MULTI_ARG_DISTINCT = False
SUPPORTS_TO_NUMBER = False
HEX_FUNC = "TO_HEX"
PROPERTIES_LOCATION = {
**generator.Generator.PROPERTIES_LOCATION,
@ -381,7 +385,6 @@ class Presto(Dialect):
exp.GroupConcat: lambda self, e: self.func(
"ARRAY_JOIN", self.func("ARRAY_AGG", e.this), e.args.get("separator")
),
exp.Hex: rename_func("TO_HEX"),
exp.If: if_sql(),
exp.ILike: no_ilike_sql,
exp.Initcap: _initcap_sql,
@ -417,7 +420,7 @@ class Presto(Dialect):
exp.StructExtract: struct_extract_sql,
exp.Table: transforms.preprocess([_unnest_sequence]),
exp.Timestamp: no_timestamp_sql,
exp.TimestampTrunc: timestamptrunc_sql,
exp.TimestampTrunc: timestamptrunc_sql(),
exp.TimeStrToDate: timestrtotime_sql,
exp.TimeStrToTime: timestrtotime_sql,
exp.TimeStrToUnix: lambda self, e: self.func(
@ -444,6 +447,14 @@ class Presto(Dialect):
[transforms.remove_within_group_for_percentiles]
),
exp.Xor: bool_xor_sql,
exp.MD5: lambda self, e: self.func(
"LOWER", self.func("TO_HEX", self.func("MD5", self.sql(e, "this")))
),
exp.MD5Digest: rename_func("MD5"),
exp.SHA: rename_func("SHA1"),
exp.SHA2: lambda self, e: self.func(
"SHA256" if e.text("length") == "256" else "SHA512", e.this
),
}
RESERVED_KEYWORDS = {
@ -615,3 +626,25 @@ class Presto(Dialect):
if kind == "VIEW" and schema.expressions:
expression.this.set("expressions", None)
return super().create_sql(expression)
def delete_sql(self, expression: exp.Delete) -> str:
"""
Presto only supports DELETE FROM for a single table without an alias, so we need
to remove the unnecessary parts. If the original DELETE statement contains more
than one table to be deleted, we can't safely map it 1-1 to a Presto statement.
"""
tables = expression.args.get("tables") or [expression.this]
if len(tables) > 1:
return super().delete_sql(expression)
table = tables[0]
expression.set("this", table)
expression.set("tables", None)
if isinstance(table, exp.Table):
table_alias = table.args.get("alias")
if table_alias:
table_alias.pop()
expression = t.cast(exp.Delete, expression.transform(unqualify_columns))
return super().delete_sql(expression)

19
sqlglot/dialects/redshift.py
View file

@ -39,6 +39,7 @@ class Redshift(Postgres):
SUPPORTS_USER_DEFINED_TYPES = False
INDEX_OFFSET = 0
COPY_PARAMS_ARE_CSV = False
HEX_LOWERCASE = True
TIME_FORMAT = "'YYYY-MM-DD HH:MI:SS'"
TIME_MAPPING = {
@ -140,6 +141,9 @@ class Redshift(Postgres):
CAN_IMPLEMENT_ARRAY_ANY = False
MULTI_ARG_DISTINCT = True
COPY_PARAMS_ARE_WRAPPED = False
HEX_FUNC = "TO_HEX"
# Redshift doesn't have `WITH` as part of their with_properties so we remove it
WITH_PROPERTIES_PREFIX = " "
TYPE_MAPPING = {
**Postgres.Generator.TYPE_MAPPING,
@ -169,6 +173,7 @@ class Redshift(Postgres):
exp.JSONExtract: json_extract_segments("JSON_EXTRACT_PATH_TEXT"),
exp.JSONExtractScalar: json_extract_segments("JSON_EXTRACT_PATH_TEXT"),
exp.GroupConcat: rename_func("LISTAGG"),
exp.Hex: lambda self, e: self.func("UPPER", self.func("TO_HEX", self.sql(e, "this"))),
exp.ParseJSON: rename_func("JSON_PARSE"),
exp.Select: transforms.preprocess(
[
@ -372,10 +377,6 @@ class Redshift(Postgres):
alias = self.expressions(expression.args.get("alias"), key="columns", flat=True)
return f"{arg} AS {alias}" if alias else arg
def with_properties(self, properties: exp.Properties) -> str:
"""Redshift doesn't have `WITH` as part of their with_properties so we remove it"""
return self.properties(properties, prefix=" ", suffix="")
def cast_sql(self, expression: exp.Cast, safe_prefix: t.Optional[str] = None) -> str:
if expression.is_type(exp.DataType.Type.JSON):
# Redshift doesn't support a JSON type, so casting to it is treated as a noop
@ -398,3 +399,13 @@ class Redshift(Postgres):
expression.append("expressions", exp.var("MAX"))
return super().datatype_sql(expression)
def alterset_sql(self, expression: exp.AlterSet) -> str:
exprs = self.expressions(expression, flat=True)
exprs = f" TABLE PROPERTIES ({exprs})" if exprs else ""
location = self.sql(expression, "location")
location = f" LOCATION {location}" if location else ""
file_format = self.expressions(expression, key="file_format", flat=True, sep=" ")
file_format = f" FILE FORMAT {file_format}" if file_format else ""
return f"SET{exprs}{location}{file_format}"

78
sqlglot/dialects/snowflake.py
View file

@ -7,6 +7,7 @@ from sqlglot.dialects.dialect import (
Dialect,
NormalizationStrategy,
binary_from_function,
build_default_decimal_type,
date_delta_sql,
date_trunc_to_time,
datestrtodate_sql,
@ -334,6 +335,7 @@ class Snowflake(Dialect):
class Parser(parser.Parser):
IDENTIFY_PIVOT_STRINGS = True
DEFAULT_SAMPLING_METHOD = "BERNOULLI"
ID_VAR_TOKENS = {
*parser.Parser.ID_VAR_TOKENS,
@ -345,6 +347,7 @@ class Snowflake(Dialect):
FUNCTIONS = {
**parser.Parser.FUNCTIONS,
"APPROX_PERCENTILE": exp.ApproxQuantile.from_arg_list,
"ARRAYAGG": exp.ArrayAgg.from_arg_list,
"ARRAY_CONSTRUCT": exp.Array.from_arg_list,
"ARRAY_CONTAINS": lambda args: exp.ArrayContains(
@ -423,7 +426,6 @@ class Snowflake(Dialect):
ALTER_PARSERS = {
**parser.Parser.ALTER_PARSERS,
"SET": lambda self: self._parse_set(tag=self._match_text_seq("TAG")),
"UNSET": lambda self: self.expression(
exp.Set,
tag=self._match_text_seq("TAG"),
@ -443,6 +445,11 @@ class Snowflake(Dialect):
"LOCATION": lambda self: self._parse_location_property(),
}
TYPE_CONVERTER = {
# https://docs.snowflake.com/en/sql-reference/data-types-numeric#number
exp.DataType.Type.DECIMAL: build_default_decimal_type(precision=38, scale=0),
}
SHOW_PARSERS = {
"SCHEMAS": _show_parser("SCHEMAS"),
"TERSE SCHEMAS": _show_parser("SCHEMAS"),
@ -475,6 +482,14 @@ class Snowflake(Dialect):
SCHEMA_KINDS = {"OBJECTS", "TABLES", "VIEWS", "SEQUENCES", "UNIQUE KEYS", "IMPORTED KEYS"}
def _parse_create(self) -> exp.Create | exp.Command:
expression = super()._parse_create()
if isinstance(expression, exp.Create) and expression.kind == "TAG":
# Replace the Table node with the enclosed Identifier
expression.this.replace(expression.this.this)
return expression
def _parse_column_ops(self, this: t.Optional[exp.Expression]) -> t.Optional[exp.Expression]:
this = super()._parse_column_ops(this)
@ -600,8 +615,8 @@ class Snowflake(Dialect):
file_format = None
pattern = None
self._match(TokenType.L_PAREN)
while self._curr and not self._match(TokenType.R_PAREN):
wrapped = self._match(TokenType.L_PAREN)
while self._curr and wrapped and not self._match(TokenType.R_PAREN):
if self._match_text_seq("FILE_FORMAT", "=>"):
file_format = self._parse_string() or super()._parse_table_parts(
is_db_reference=is_db_reference
@ -681,14 +696,22 @@ class Snowflake(Dialect):
return self.expression(exp.LocationProperty, this=self._parse_location_path())
def _parse_file_location(self) -> t.Optional[exp.Expression]:
return self._parse_table_parts()
# Parse either a subquery or a staged file
return (
self._parse_select(table=True)
if self._match(TokenType.L_PAREN, advance=False)
else self._parse_table_parts()
)
def _parse_location_path(self) -> exp.Var:
parts = [self._advance_any(ignore_reserved=True)]
# We avoid consuming a comma token because external tables like @foo and @bar
# can be joined in a query with a comma separator.
while self._is_connected() and not self._match(TokenType.COMMA, advance=False):
# can be joined in a query with a comma separator, as well as closing paren
# in case of subqueries
while self._is_connected() and not self._match_set(
(TokenType.COMMA, TokenType.R_PAREN), advance=False
):
parts.append(self._advance_any(ignore_reserved=True))
return exp.var("".join(part.text for part in parts if part))
@ -713,12 +736,12 @@ class Snowflake(Dialect):
"NCHAR VARYING": TokenType.VARCHAR,
"PUT": TokenType.COMMAND,
"REMOVE": TokenType.COMMAND,
"RENAME": TokenType.REPLACE,
"RM": TokenType.COMMAND,
"SAMPLE": TokenType.TABLE_SAMPLE,
"SQL_DOUBLE": TokenType.DOUBLE,
"SQL_VARCHAR": TokenType.VARCHAR,
"STORAGE INTEGRATION": TokenType.STORAGE_INTEGRATION,
"TAG": TokenType.TAG,
"TIMESTAMP_TZ": TokenType.TIMESTAMPTZ,
"TOP": TokenType.TOP,
}
@ -748,6 +771,7 @@ class Snowflake(Dialect):
STRUCT_DELIMITER = ("(", ")")
COPY_PARAMS_ARE_WRAPPED = False
COPY_PARAMS_EQ_REQUIRED = True
STAR_EXCEPT = "EXCLUDE"
TRANSFORMS = {
**generator.Generator.TRANSFORMS,
@ -818,7 +842,7 @@ class Snowflake(Dialect):
exp.TimestampDiff: lambda self, e: self.func(
"TIMESTAMPDIFF", e.unit, e.expression, e.this
),
exp.TimestampTrunc: timestamptrunc_sql,
exp.TimestampTrunc: timestamptrunc_sql(),
exp.TimeStrToTime: timestrtotime_sql,
exp.TimeToStr: lambda self, e: self.func(
"TO_CHAR", exp.cast(e.this, exp.DataType.Type.TIMESTAMP), self.format_time(e)
@ -850,11 +874,6 @@ class Snowflake(Dialect):
exp.DataType.Type.STRUCT: "OBJECT",
}
STAR_MAPPING = {
"except": "EXCLUDE",
"replace": "RENAME",
}
PROPERTIES_LOCATION = {
**generator.Generator.PROPERTIES_LOCATION,
exp.SetProperty: exp.Properties.Location.UNSUPPORTED,
@ -862,9 +881,15 @@ class Snowflake(Dialect):
}
UNSUPPORTED_VALUES_EXPRESSIONS = {
exp.Map,
exp.StarMap,
exp.Struct,
exp.VarMap,
}
def with_properties(self, properties: exp.Properties) -> str:
return self.properties(properties, wrapped=False, prefix=self.sep(""), sep=" ")
def values_sql(self, expression: exp.Values, values_as_table: bool = True) -> str:
if expression.find(*self.UNSUPPORTED_VALUES_EXPRESSIONS):
values_as_table = False
@ -1019,9 +1044,6 @@ class Snowflake(Dialect):
this = self.sql(expression, "this")
return f"SWAP WITH {this}"
def with_properties(self, properties: exp.Properties) -> str:
return self.properties(properties, wrapped=False, prefix=self.seg(""), sep=" ")
def cluster_sql(self, expression: exp.Cluster) -> str:
return f"CLUSTER BY ({self.expressions(expression, flat=True)})"
@ -1041,10 +1063,22 @@ class Snowflake(Dialect):
return self.func("OBJECT_CONSTRUCT", *flatten(zip(keys, values)))
def copyparameter_sql(self, expression: exp.CopyParameter) -> str:
option = self.sql(expression, "this").upper()
if option == "FILE_FORMAT":
values = self.expressions(expression, key="expression", flat=True, sep=" ")
return f"{option} = ({values})"
def approxquantile_sql(self, expression: exp.ApproxQuantile) -> str:
if expression.args.get("weight") or expression.args.get("accuracy"):
self.unsupported(
"APPROX_PERCENTILE with weight and/or accuracy arguments are not supported in Snowflake"
)
return super().copyparameter_sql(expression)
return self.func("APPROX_PERCENTILE", expression.this, expression.args.get("quantile"))
def alterset_sql(self, expression: exp.AlterSet) -> str:
exprs = self.expressions(expression, flat=True)
exprs = f" {exprs}" if exprs else ""
file_format = self.expressions(expression, key="file_format", flat=True, sep=" ")
file_format = f" STAGE_FILE_FORMAT = ({file_format})" if file_format else ""
copy_options = self.expressions(expression, key="copy_options", flat=True, sep=" ")
copy_options = f" STAGE_COPY_OPTIONS = ({copy_options})" if copy_options else ""
tag = self.expressions(expression, key="tag", flat=True)
tag = f" TAG {tag}" if tag else ""
return f"SET{exprs}{file_format}{copy_options}{tag}"

3
sqlglot/dialects/spark.py
View file

@ -96,6 +96,9 @@ class Spark(Spark2):
TRANSFORMS = {
**Spark2.Generator.TRANSFORMS,
exp.ArrayConstructCompact: lambda self, e: self.func(
"ARRAY_COMPACT", self.func("ARRAY", *e.expressions)
),
exp.Create: preprocess(
[
remove_unique_constraints,

71
sqlglot/dialects/tsql.py
View file

@ -3,6 +3,7 @@ from __future__ import annotations
import datetime
import re
import typing as t
from functools import partial
from sqlglot import exp, generator, parser, tokens, transforms
from sqlglot.dialects.dialect import (
@ -211,7 +212,8 @@ def _string_agg_sql(self: TSQL.Generator, expression: exp.GroupConcat) -> str:
if isinstance(expression.this, exp.Order):
if expression.this.this:
this = expression.this.this.pop()
order = f" WITHIN GROUP ({self.sql(expression.this)[1:]})" # Order has a leading space
# Order has a leading space
order = f" WITHIN GROUP ({self.sql(expression.this)[1:]})"
separator = expression.args.get("separator") or exp.Literal.string(",")
return f"STRING_AGG({self.format_args(this, separator)}){order}"
@ -451,11 +453,14 @@ class TSQL(Dialect):
**tokens.Tokenizer.KEYWORDS,
"DATETIME2": TokenType.DATETIME,
"DATETIMEOFFSET": TokenType.TIMESTAMPTZ,
"DECLARE": TokenType.COMMAND,
"DECLARE": TokenType.DECLARE,
"EXEC": TokenType.COMMAND,
"FOR SYSTEM_TIME": TokenType.TIMESTAMP_SNAPSHOT,
"IMAGE": TokenType.IMAGE,
"MONEY": TokenType.MONEY,
"NTEXT": TokenType.TEXT,
"OPTION": TokenType.OPTION,
"OUTPUT": TokenType.RETURNING,
"PRINT": TokenType.COMMAND,
"PROC": TokenType.PROCEDURE,
"REAL": TokenType.FLOAT,
@ -463,17 +468,17 @@ class TSQL(Dialect):
"SMALLDATETIME": TokenType.DATETIME,
"SMALLMONEY": TokenType.SMALLMONEY,
"SQL_VARIANT": TokenType.VARIANT,
"SYSTEM_USER": TokenType.CURRENT_USER,
"TOP": TokenType.TOP,
"TIMESTAMP": TokenType.ROWVERSION,
"TINYINT": TokenType.UTINYINT,
"UNIQUEIDENTIFIER": TokenType.UNIQUEIDENTIFIER,
"UPDATE STATISTICS": TokenType.COMMAND,
"XML": TokenType.XML,
"OUTPUT": TokenType.RETURNING,
"SYSTEM_USER": TokenType.CURRENT_USER,
"FOR SYSTEM_TIME": TokenType.TIMESTAMP_SNAPSHOT,
"OPTION": TokenType.OPTION,
}
COMMANDS = {*tokens.Tokenizer.COMMANDS, TokenType.END}
class Parser(parser.Parser):
SET_REQUIRES_ASSIGNMENT_DELIMITER = False
LOG_DEFAULTS_TO_LN = True
@ -526,7 +531,7 @@ class TSQL(Dialect):
STATEMENT_PARSERS = {
**parser.Parser.STATEMENT_PARSERS,
TokenType.END: lambda self: self._parse_command(),
TokenType.DECLARE: lambda self: self._parse_declare(),
}
def _parse_options(self) -> t.Optional[t.List[exp.Expression]]:
@ -711,6 +716,32 @@ class TSQL(Dialect):
return partition
def _parse_declare(self) -> exp.Declare | exp.Command:
index = self._index
expressions = self._try_parse(partial(self._parse_csv, self._parse_declareitem))
if not expressions or self._curr:
self._retreat(index)
return self._parse_as_command(self._prev)
return self.expression(exp.Declare, expressions=expressions)
def _parse_declareitem(self) -> t.Optional[exp.DeclareItem]:
var = self._parse_id_var()
if not var:
return None
value = None
self._match(TokenType.ALIAS)
if self._match(TokenType.TABLE):
data_type = self._parse_schema()
else:
data_type = self._parse_types()
if self._match(TokenType.EQ):
value = self._parse_bitwise()
return self.expression(exp.DeclareItem, this=var, kind=data_type, default=value)
class Generator(generator.Generator):
LIMIT_IS_TOP = True
QUERY_HINTS = False
@ -753,11 +784,12 @@ class TSQL(Dialect):
exp.DataType.Type.DATETIME: "DATETIME2",
exp.DataType.Type.DOUBLE: "FLOAT",
exp.DataType.Type.INT: "INTEGER",
exp.DataType.Type.ROWVERSION: "ROWVERSION",
exp.DataType.Type.TEXT: "VARCHAR(MAX)",
exp.DataType.Type.TIMESTAMP: "DATETIME2",
exp.DataType.Type.TIMESTAMPTZ: "DATETIMEOFFSET",
exp.DataType.Type.UTINYINT: "TINYINT",
exp.DataType.Type.VARIANT: "SQL_VARIANT",
exp.DataType.Type.ROWVERSION: "ROWVERSION",
}
TYPE_MAPPING.pop(exp.DataType.Type.NCHAR)
@ -920,6 +952,10 @@ class TSQL(Dialect):
def create_sql(self, expression: exp.Create) -> str:
kind = expression.kind
exists = expression.args.pop("exists", None)
if kind == "VIEW":
expression.this.set("catalog", None)
sql = super().create_sql(expression)
like_property = expression.find(exp.LikeProperty)
@ -1061,3 +1097,22 @@ class TSQL(Dialect):
if isinstance(action, exp.RenameTable):
return f"EXEC sp_rename '{self.sql(expression.this)}', '{action.this.name}'"
return super().altertable_sql(expression)
def drop_sql(self, expression: exp.Drop) -> str:
if expression.args["kind"] == "VIEW":
expression.this.set("catalog", None)
return super().drop_sql(expression)
def declare_sql(self, expression: exp.Declare) -> str:
return f"DECLARE {self.expressions(expression, flat=True)}"
def declareitem_sql(self, expression: exp.DeclareItem) -> str:
variable = self.sql(expression, "this")
default = self.sql(expression, "default")
default = f" = {default}" if default else ""
kind = self.sql(expression, "kind")
if isinstance(expression.args.get("kind"), exp.Schema):
kind = f"TABLE {kind}"
return f"{variable} AS {kind}{default}"

4
sqlglot/executor/env.py
View file

@ -240,5 +240,7 @@ ENV = {
for x in range(0, len(args), 2)
if (args[x + 1] is not None and args[x] is not None)
},
"UNIXTOTIME": null_if_any(lambda arg: datetime.datetime.utcfromtimestamp(arg)),
"UNIXTOTIME": null_if_any(
lambda arg: datetime.datetime.fromtimestamp(arg, datetime.timezone.utc)
),
}

89
sqlglot/expressions.py
View file

@ -1445,6 +1445,14 @@ class Pragma(Expression):
pass
class Declare(Expression):
arg_types = {"expressions": True}
class DeclareItem(Expression):
arg_types = {"this": True, "kind": True, "default": False}
class Set(Expression):
arg_types = {"expressions": False, "unset": False, "tag": False}
@ -1520,6 +1528,10 @@ class CTE(DerivedTable):
}
class ProjectionDef(Expression):
arg_types = {"this": True, "expression": True}
class TableAlias(Expression):
arg_types = {"this": False, "columns": False}
@ -1623,6 +1635,29 @@ class AlterColumn(Expression):
}
# https://docs.aws.amazon.com/redshift/latest/dg/r_ALTER_TABLE.html
class AlterDistStyle(Expression):
pass
class AlterSortKey(Expression):
arg_types = {"this": False, "expressions": False, "compound": False}
class AlterSet(Expression):
arg_types = {
"expressions": False,
"option": False,
"tablespace": False,
"access_method": False,
"file_format": False,
"copy_options": False,
"tag": False,
"location": False,
"serde": False,
}
class RenameColumn(Expression):
arg_types = {"this": True, "to": True, "exists": False}
@ -1939,6 +1974,7 @@ class Drop(Expression):
"cascade": False,
"constraints": False,
"purge": False,
"cluster": False,
}
@ -2177,6 +2213,11 @@ class PartitionRange(Expression):
arg_types = {"this": True, "expression": True}
# https://clickhouse.com/docs/en/sql-reference/statements/alter/partition#how-to-set-partition-expression
class PartitionId(Expression):
pass
class Fetch(Expression):
arg_types = {
"direction": False,
@ -2422,6 +2463,10 @@ class Property(Expression):
arg_types = {"this": True, "value": True}
class AllowedValuesProperty(Expression):
arg_types = {"expressions": True}
class AlgorithmProperty(Property):
arg_types = {"this": True}
@ -2475,6 +2520,10 @@ class DataBlocksizeProperty(Property):
}
class DataDeletionProperty(Property):
arg_types = {"on": True, "filter_col": False, "retention_period": False}
class DefinerProperty(Property):
arg_types = {"this": True}
@ -2651,7 +2700,11 @@ class RemoteWithConnectionModelProperty(Property):
class ReturnsProperty(Property):
arg_types = {"this": True, "is_table": False, "table": False}
arg_types = {"this": False, "is_table": False, "table": False, "null": False}
class StrictProperty(Property):
arg_types = {}
class RowFormatProperty(Property):
@ -2697,7 +2750,7 @@ class SchemaCommentProperty(Property):
class SerdeProperties(Property):
arg_types = {"expressions": True}
arg_types = {"expressions": True, "with": False}
class SetProperty(Property):
@ -2766,8 +2819,13 @@ class WithJournalTableProperty(Property):
class WithSystemVersioningProperty(Property):
# this -> history table name, expression -> data consistency check
arg_types = {"this": False, "expression": False}
arg_types = {
"on": False,
"this": False,
"data_consistency": False,
"retention_period": False,
"with": True,
}
class Properties(Expression):
@ -3801,7 +3859,7 @@ class Where(Expression):
class Star(Expression):
arg_types = {"except": False, "replace": False}
arg_types = {"except": False, "replace": False, "rename": False}
@property
def name(self) -> str:
@ -4175,6 +4233,7 @@ class AlterTable(Expression):
"exists": False,
"only": False,
"options": False,
"cluster": False,
}
@ -4186,6 +4245,11 @@ class DropPartition(Expression):
arg_types = {"expressions": True, "exists": False}
# https://clickhouse.com/docs/en/sql-reference/statements/alter/partition#replace-partition
class ReplacePartition(Expression):
arg_types = {"expression": True, "source": True}
# Binary expressions like (ADD a b)
class Binary(Condition):
arg_types = {"this": True, "expression": True}
@ -4738,6 +4802,11 @@ class ArrayConcat(Func):
is_var_len_args = True
class ArrayConstructCompact(Func):
arg_types = {"expressions": True}
is_var_len_args = True
class ArrayContains(Binary, Func):
pass
@ -5172,6 +5241,10 @@ class Hex(Func):
pass
class LowerHex(Hex):
pass
class Xor(Connector, Func):
arg_types = {"this": False, "expression": False, "expressions": False}
@ -5902,6 +5975,12 @@ class NextValueFor(Func):
arg_types = {"this": True, "order": False}
# Refers to a trailing semi-colon. This is only used to preserve trailing comments
# select 1; -- my comment
class Semicolon(Expression):
arg_types = {}
def _norm_arg(arg):
return arg.lower() if type(arg) is str else arg

191
sqlglot/generator.py
View file

@ -8,7 +8,7 @@ from functools import reduce
from sqlglot import exp
from sqlglot.errors import ErrorLevel, UnsupportedError, concat_messages
from sqlglot.helper import apply_index_offset, csv, seq_get
from sqlglot.helper import apply_index_offset, csv, name_sequence, seq_get
from sqlglot.jsonpath import ALL_JSON_PATH_PARTS, JSON_PATH_PART_TRANSFORMS
from sqlglot.time import format_time
from sqlglot.tokens import TokenType
@ -74,6 +74,8 @@ class Generator(metaclass=_Generator):
TRANSFORMS: t.Dict[t.Type[exp.Expression], t.Callable[..., str]] = {
**JSON_PATH_PART_TRANSFORMS,
exp.AllowedValuesProperty: lambda self,
e: f"ALLOWED_VALUES {self.expressions(e, flat=True)}",
exp.AutoRefreshProperty: lambda self, e: f"AUTO REFRESH {self.sql(e, 'this')}",
exp.BackupProperty: lambda self, e: f"BACKUP {self.sql(e, 'this')}",
exp.CaseSpecificColumnConstraint: lambda _,
@ -123,7 +125,9 @@ class Generator(metaclass=_Generator):
exp.PathColumnConstraint: lambda self, e: f"PATH {self.sql(e, 'this')}",
exp.RemoteWithConnectionModelProperty: lambda self,
e: f"REMOTE WITH CONNECTION {self.sql(e, 'this')}",
exp.ReturnsProperty: lambda self, e: self.naked_property(e),
exp.ReturnsProperty: lambda self, e: (
"RETURNS NULL ON NULL INPUT" if e.args.get("null") else self.naked_property(e)
),
exp.SampleProperty: lambda self, e: f"SAMPLE BY {self.sql(e, 'this')}",
exp.SetConfigProperty: lambda self, e: self.sql(e, "this"),
exp.SetProperty: lambda _, e: f"{'MULTI' if e.args.get('multi') else ''}SET",
@ -133,6 +137,7 @@ class Generator(metaclass=_Generator):
exp.SqlSecurityProperty: lambda _,
e: f"SQL SECURITY {'DEFINER' if e.args.get('definer') else 'INVOKER'}",
exp.StabilityProperty: lambda _, e: e.name,
exp.StrictProperty: lambda *_: "STRICT",
exp.TemporaryProperty: lambda *_: "TEMPORARY",
exp.TitleColumnConstraint: lambda self, e: f"TITLE {self.sql(e, 'this')}",
exp.Timestamp: lambda self, e: self.func("TIMESTAMP", e.this, e.expression),
@ -351,6 +356,15 @@ class Generator(metaclass=_Generator):
# Whether the conditional TRY(expression) function is supported
TRY_SUPPORTED = True
# The keyword to use when generating a star projection with excluded columns
STAR_EXCEPT = "EXCEPT"
# The HEX function name
HEX_FUNC = "HEX"
# The keywords to use when prefixing & separating WITH based properties
WITH_PROPERTIES_PREFIX = "WITH"
TYPE_MAPPING = {
exp.DataType.Type.NCHAR: "CHAR",
exp.DataType.Type.NVARCHAR: "VARCHAR",
@ -364,11 +378,6 @@ class Generator(metaclass=_Generator):
exp.DataType.Type.ROWVERSION: "VARBINARY",
}
STAR_MAPPING = {
"except": "EXCEPT",
"replace": "REPLACE",
}
TIME_PART_SINGULARS = {
"MICROSECONDS": "MICROSECOND",
"SECONDS": "SECOND",
@ -401,6 +410,7 @@ class Generator(metaclass=_Generator):
NAMED_PLACEHOLDER_TOKEN = ":"
PROPERTIES_LOCATION = {
exp.AllowedValuesProperty: exp.Properties.Location.POST_SCHEMA,
exp.AlgorithmProperty: exp.Properties.Location.POST_CREATE,
exp.AutoIncrementProperty: exp.Properties.Location.POST_SCHEMA,
exp.AutoRefreshProperty: exp.Properties.Location.POST_SCHEMA,
@ -413,6 +423,7 @@ class Generator(metaclass=_Generator):
exp.Cluster: exp.Properties.Location.POST_SCHEMA,
exp.ClusteredByProperty: exp.Properties.Location.POST_SCHEMA,
exp.DataBlocksizeProperty: exp.Properties.Location.POST_NAME,
exp.DataDeletionProperty: exp.Properties.Location.POST_SCHEMA,
exp.DefinerProperty: exp.Properties.Location.POST_CREATE,
exp.DictRange: exp.Properties.Location.POST_SCHEMA,
exp.DictProperty: exp.Properties.Location.POST_SCHEMA,
@ -466,6 +477,7 @@ class Generator(metaclass=_Generator):
exp.SqlReadWriteProperty: exp.Properties.Location.POST_SCHEMA,
exp.SqlSecurityProperty: exp.Properties.Location.POST_CREATE,
exp.StabilityProperty: exp.Properties.Location.POST_SCHEMA,
exp.StrictProperty: exp.Properties.Location.POST_SCHEMA,
exp.TemporaryProperty: exp.Properties.Location.POST_CREATE,
exp.ToTableProperty: exp.Properties.Location.POST_SCHEMA,
exp.TransientProperty: exp.Properties.Location.POST_CREATE,
@ -539,6 +551,7 @@ class Generator(metaclass=_Generator):
"unsupported_messages",
"_escaped_quote_end",
"_escaped_identifier_end",
"_next_name",
)
def __init__(
@ -584,6 +597,8 @@ class Generator(metaclass=_Generator):
self.dialect.tokenizer_class.IDENTIFIER_ESCAPES[0] + self.dialect.IDENTIFIER_END
)
self._next_name = name_sequence("_t")
def generate(self, expression: exp.Expression, copy: bool = True) -> str:
"""
Generates the SQL string corresponding to the given syntax tree.
@ -687,15 +702,15 @@ class Generator(metaclass=_Generator):
return f"{sql} {comments_sql}"
def wrap(self, expression: exp.Expression | str) -> str:
this_sql = self.indent(
(
self.sql(expression)
if isinstance(expression, exp.UNWRAPPED_QUERIES)
else self.sql(expression, "this")
),
level=1,
pad=0,
this_sql = (
self.sql(expression)
if isinstance(expression, exp.UNWRAPPED_QUERIES)
else self.sql(expression, "this")
)
if not this_sql:
return "()"
this_sql = self.indent(this_sql, level=1, pad=0)
return f"({self.sep('')}{this_sql}{self.seg(')', sep='')}"
def no_identify(self, func: t.Callable[..., str], *args, **kwargs) -> str:
@ -720,7 +735,7 @@ class Generator(metaclass=_Generator):
skip_first: bool = False,
skip_last: bool = False,
) -> str:
if not self.pretty:
if not self.pretty or not sql:
return sql
pad = self.pad if pad is None else pad
@ -951,6 +966,12 @@ class Generator(metaclass=_Generator):
)
)
if properties_locs.get(exp.Properties.Location.POST_SCHEMA):
properties_sql = self.sep() + properties_sql
elif not self.pretty:
# Standalone POST_WITH properties need a leading whitespace in non-pretty mode
properties_sql = f" {properties_sql}"
begin = " BEGIN" if expression.args.get("begin") else ""
end = " END" if expression.args.get("end") else ""
@ -1095,7 +1116,7 @@ class Generator(metaclass=_Generator):
self.unsupported("Named columns are not supported in table alias.")
if not alias and not self.dialect.UNNEST_COLUMN_ONLY:
alias = "_t"
alias = self._next_name()
return f"{alias}{columns}"
@ -1208,12 +1229,14 @@ class Generator(metaclass=_Generator):
expressions = f" ({expressions})" if expressions else ""
kind = expression.args["kind"]
exists_sql = " IF EXISTS " if expression.args.get("exists") else " "
on_cluster = self.sql(expression, "cluster")
on_cluster = f" {on_cluster}" if on_cluster else ""
temporary = " TEMPORARY" if expression.args.get("temporary") else ""
materialized = " MATERIALIZED" if expression.args.get("materialized") else ""
cascade = " CASCADE" if expression.args.get("cascade") else ""
constraints = " CONSTRAINTS" if expression.args.get("constraints") else ""
purge = " PURGE" if expression.args.get("purge") else ""
return f"DROP{temporary}{materialized} {kind}{exists_sql}{this}{expressions}{cascade}{constraints}{purge}"
return f"DROP{temporary}{materialized} {kind}{exists_sql}{this}{on_cluster}{expressions}{cascade}{constraints}{purge}"
def except_sql(self, expression: exp.Except) -> str:
return self.set_operations(expression)
@ -1296,6 +1319,19 @@ class Generator(metaclass=_Generator):
text = f"{self.dialect.IDENTIFIER_START}{text}{self.dialect.IDENTIFIER_END}"
return text
def hex_sql(self, expression: exp.Hex) -> str:
text = self.func(self.HEX_FUNC, self.sql(expression, "this"))
if self.dialect.HEX_LOWERCASE:
text = self.func("LOWER", text)
return text
def lowerhex_sql(self, expression: exp.LowerHex) -> str:
text = self.func(self.HEX_FUNC, self.sql(expression, "this"))
if not self.dialect.HEX_LOWERCASE:
text = self.func("LOWER", text)
return text
def inputoutputformat_sql(self, expression: exp.InputOutputFormat) -> str:
input_format = self.sql(expression, "input_format")
input_format = f"INPUTFORMAT {input_format}" if input_format else ""
@ -1321,13 +1357,17 @@ class Generator(metaclass=_Generator):
elif p_loc == exp.Properties.Location.POST_SCHEMA:
root_properties.append(p)
return self.root_properties(
exp.Properties(expressions=root_properties)
) + self.with_properties(exp.Properties(expressions=with_properties))
root_props = self.root_properties(exp.Properties(expressions=root_properties))
with_props = self.with_properties(exp.Properties(expressions=with_properties))
if root_props and with_props and not self.pretty:
with_props = " " + with_props
return root_props + with_props
def root_properties(self, properties: exp.Properties) -> str:
if properties.expressions:
return self.sep() + self.expressions(properties, indent=False, sep=" ")
return self.expressions(properties, indent=False, sep=" ")
return ""
def properties(
@ -1346,7 +1386,7 @@ class Generator(metaclass=_Generator):
return ""
def with_properties(self, properties: exp.Properties) -> str:
return self.properties(properties, prefix=self.seg("WITH"))
return self.properties(properties, prefix=self.seg(self.WITH_PROPERTIES_PREFIX, sep=""))
def locate_properties(self, properties: exp.Properties) -> t.DefaultDict:
properties_locs = defaultdict(list)
@ -1514,19 +1554,25 @@ class Generator(metaclass=_Generator):
return f"{data_sql}{statistics_sql}"
def withsystemversioningproperty_sql(self, expression: exp.WithSystemVersioningProperty) -> str:
sql = "WITH(SYSTEM_VERSIONING=ON"
this = self.sql(expression, "this")
this = f"HISTORY_TABLE={this}" if this else ""
data_consistency: t.Optional[str] = self.sql(expression, "data_consistency")
data_consistency = (
f"DATA_CONSISTENCY_CHECK={data_consistency}" if data_consistency else None
)
retention_period: t.Optional[str] = self.sql(expression, "retention_period")
retention_period = (
f"HISTORY_RETENTION_PERIOD={retention_period}" if retention_period else None
)
if expression.this:
history_table = self.sql(expression, "this")
sql = f"{sql}(HISTORY_TABLE={history_table}"
if this:
on_sql = self.func("ON", this, data_consistency, retention_period)
else:
on_sql = "ON" if expression.args.get("on") else "OFF"
if expression.expression:
data_consistency_check = self.sql(expression, "expression")
sql = f"{sql}, DATA_CONSISTENCY_CHECK={data_consistency_check}"
sql = f"SYSTEM_VERSIONING={on_sql}"
sql = f"{sql})"
return f"{sql})"
return f"WITH({sql})" if expression.args.get("with") else sql
def insert_sql(self, expression: exp.Insert) -> str:
hint = self.sql(expression, "hint")
@ -2300,10 +2346,12 @@ class Generator(metaclass=_Generator):
def star_sql(self, expression: exp.Star) -> str:
except_ = self.expressions(expression, key="except", flat=True)
except_ = f"{self.seg(self.STAR_MAPPING['except'])} ({except_})" if except_ else ""
except_ = f"{self.seg(self.STAR_EXCEPT)} ({except_})" if except_ else ""
replace = self.expressions(expression, key="replace", flat=True)
replace = f"{self.seg(self.STAR_MAPPING['replace'])} ({replace})" if replace else ""
return f"*{except_}{replace}"
replace = f"{self.seg('REPLACE')} ({replace})" if replace else ""
rename = self.expressions(expression, key="rename", flat=True)
rename = f"{self.seg('RENAME')} ({rename})" if rename else ""
return f"*{except_}{replace}{rename}"
def parameter_sql(self, expression: exp.Parameter) -> str:
this = self.sql(expression, "this")
@ -2843,9 +2891,10 @@ class Generator(metaclass=_Generator):
stack.append(self.expressions(expression, sep=f" {op} "))
else:
stack.append(expression.right)
if expression.comments:
if expression.comments and self.comments:
for comment in expression.comments:
op += f" /*{self.pad_comment(comment)}*/"
if comment:
op += f" /*{self.pad_comment(comment)}*/"
stack.extend((op, expression.left))
return op
@ -2978,6 +3027,19 @@ class Generator(metaclass=_Generator):
return f"ALTER COLUMN {this} DROP DEFAULT"
def alterdiststyle_sql(self, expression: exp.AlterDistStyle) -> str:
this = self.sql(expression, "this")
if not isinstance(expression.this, exp.Var):
this = f"KEY DISTKEY {this}"
return f"ALTER DISTSTYLE {this}"
def altersortkey_sql(self, expression: exp.AlterSortKey) -> str:
compound = " COMPOUND" if expression.args.get("compound") else ""
this = self.sql(expression, "this")
expressions = self.expressions(expression, flat=True)
expressions = f"({expressions})" if expressions else ""
return f"ALTER{compound} SORTKEY {this or expressions}"
def renametable_sql(self, expression: exp.RenameTable) -> str:
if not self.RENAME_TABLE_WITH_DB:
# Remove db from tables
@ -2993,6 +3055,10 @@ class Generator(metaclass=_Generator):
new_column = self.sql(expression, "to")
return f"RENAME COLUMN{exists} {old_column} TO {new_column}"
def alterset_sql(self, expression: exp.AlterSet) -> str:
exprs = self.expressions(expression, flat=True)
return f"SET {exprs}"
def altertable_sql(self, expression: exp.AlterTable) -> str:
actions = expression.args["actions"]
@ -3006,10 +3072,12 @@ class Generator(metaclass=_Generator):
actions = self.expressions(expression, key="actions", flat=True)
exists = " IF EXISTS" if expression.args.get("exists") else ""
on_cluster = self.sql(expression, "cluster")
on_cluster = f" {on_cluster}" if on_cluster else ""
only = " ONLY" if expression.args.get("only") else ""
options = self.expressions(expression, key="options")
options = f", {options}" if options else ""
return f"ALTER TABLE{exists}{only} {self.sql(expression, 'this')} {actions}{options}"
return f"ALTER TABLE{exists}{only} {self.sql(expression, 'this')}{on_cluster} {actions}{options}"
def add_column_sql(self, expression: exp.AlterTable) -> str:
if self.ALTER_TABLE_INCLUDE_COLUMN_KEYWORD:
@ -3781,6 +3849,11 @@ class Generator(metaclass=_Generator):
def copyparameter_sql(self, expression: exp.CopyParameter) -> str:
option = self.sql(expression, "this")
if option.upper() == "FILE_FORMAT":
values = self.expressions(expression, key="expression", flat=True, sep=" ")
return f"{option} = ({values})"
value = self.sql(expression, "expression")
if not value:
@ -3802,7 +3875,6 @@ class Generator(metaclass=_Generator):
credentials = f"CREDENTIALS = ({credentials})" if credentials else ""
storage = self.sql(expression, "storage")
storage = f" {storage}" if storage else ""
encryption = self.expressions(expression, key="encryption", flat=True, sep=" ")
encryption = f" ENCRYPTION = ({encryption})" if encryption else ""
@ -3820,13 +3892,40 @@ class Generator(metaclass=_Generator):
this = f" INTO {this}" if self.COPY_HAS_INTO_KEYWORD else f" {this}"
credentials = self.sql(expression, "credentials")
credentials = f" {credentials}" if credentials else ""
kind = " FROM " if expression.args.get("kind") else " TO "
credentials = self.seg(credentials) if credentials else ""
kind = self.seg("FROM" if expression.args.get("kind") else "TO")
files = self.expressions(expression, key="files", flat=True)
sep = ", " if self.dialect.COPY_PARAMS_ARE_CSV else " "
params = self.expressions(expression, key="params", flat=True, sep=sep)
if params:
params = f" WITH ({params})" if self.COPY_PARAMS_ARE_WRAPPED else f" {params}"
params = self.expressions(
expression,
key="params",
sep=sep,
new_line=True,
skip_last=True,
skip_first=True,
indent=self.COPY_PARAMS_ARE_WRAPPED,
)
return f"COPY{this}{kind}{files}{credentials}{params}"
if params:
if self.COPY_PARAMS_ARE_WRAPPED:
params = f" WITH ({params})"
elif not self.pretty:
params = f" {params}"
return f"COPY{this}{kind} {files}{credentials}{params}"
def semicolon_sql(self, expression: exp.Semicolon) -> str:
return ""
def datadeletionproperty_sql(self, expression: exp.DataDeletionProperty) -> str:
on_sql = "ON" if expression.args.get("on") else "OFF"
filter_col: t.Optional[str] = self.sql(expression, "filter_column")
filter_col = f"FILTER_COLUMN={filter_col}" if filter_col else None
retention_period: t.Optional[str] = self.sql(expression, "retention_period")
retention_period = f"RETENTION_PERIOD={retention_period}" if retention_period else None
if filter_col or retention_period:
on_sql = self.func("ON", filter_col, retention_period)
return f"DATA_DELETION={on_sql}"

8
sqlglot/lineage.py
View file

@ -229,7 +229,9 @@ def to_node(
for source in scope.sources.values():
if isinstance(source, Scope):
source = source.expression
node.downstream.append(Node(name=select.sql(), source=source, expression=source))
node.downstream.append(
Node(name=select.sql(comments=False), source=source, expression=source)
)
# Find all columns that went into creating this one to list their lineage nodes.
source_columns = set(find_all_in_scope(select, exp.Column))
@ -278,7 +280,9 @@ def to_node(
# it means this column's lineage is unknown. This can happen if the definition of a source used in a query
# is not passed into the `sources` map.
source = source or exp.Placeholder()
node.downstream.append(Node(name=c.sql(), source=source, expression=source))
node.downstream.append(
Node(name=c.sql(comments=False), source=source, expression=source)
)
return node

31
sqlglot/optimizer/unnest_subqueries.py
View file

@ -1,6 +1,6 @@
from sqlglot import exp
from sqlglot.helper import name_sequence
from sqlglot.optimizer.scope import ScopeType, traverse_scope
from sqlglot.optimizer.scope import ScopeType, find_in_scope, traverse_scope
def unnest_subqueries(expression):
@ -64,7 +64,7 @@ def unnest(select, parent_select, next_alias_name):
(not clause or clause_parent_select is not parent_select)
and (
parent_select.args.get("group")
or any(projection.find(exp.AggFunc) for projection in parent_select.selects)
or any(find_in_scope(select, exp.AggFunc) for select in parent_select.selects)
)
):
column = exp.Max(this=column)
@ -101,7 +101,7 @@ def unnest(select, parent_select, next_alias_name):
if group:
if {value.this} != set(group.expressions):
select = (
exp.select(exp.column(value.alias, "_q"))
exp.select(exp.alias_(exp.column(value.alias, "_q"), value.alias))
.from_(select.subquery("_q", copy=False), copy=False)
.group_by(exp.column(value.alias, "_q"), copy=False)
)
@ -152,7 +152,9 @@ def decorrelate(select, parent_select, external_columns, next_alias_name):
return
is_subquery_projection = any(
node is select.parent for node in parent_select.selects if isinstance(node, exp.Subquery)
node is select.parent
for node in map(lambda s: s.unalias(), parent_select.selects)
if isinstance(node, exp.Subquery)
)
value = select.selects[0]
@ -200,19 +202,25 @@ def decorrelate(select, parent_select, external_columns, next_alias_name):
alias = exp.column(value.alias, table_alias)
other = _other_operand(parent_predicate)
op_type = type(parent_predicate.parent) if parent_predicate else None
if isinstance(parent_predicate, exp.Exists):
alias = exp.column(list(key_aliases.values())[0], table_alias)
parent_predicate = _replace(parent_predicate, f"NOT {alias} IS NULL")
elif isinstance(parent_predicate, exp.All):
assert issubclass(op_type, exp.Binary)
predicate = op_type(this=other, expression=exp.column("_x"))
parent_predicate = _replace(
parent_predicate.parent, f"ARRAY_ALL({alias}, _x -> _x = {other})"
parent_predicate.parent, f"ARRAY_ALL({alias}, _x -> {predicate})"
)
elif isinstance(parent_predicate, exp.Any):
assert issubclass(op_type, exp.Binary)
if value.this in group_by:
parent_predicate = _replace(parent_predicate.parent, f"{other} = {alias}")
predicate = op_type(this=other, expression=alias)
parent_predicate = _replace(parent_predicate.parent, predicate)
else:
parent_predicate = _replace(parent_predicate, f"ARRAY_ANY({alias}, _x -> _x = {other})")
predicate = op_type(this=other, expression=exp.column("_x"))
parent_predicate = _replace(parent_predicate, f"ARRAY_ANY({alias}, _x -> {predicate})")
elif isinstance(parent_predicate, exp.In):
if value.this in group_by:
parent_predicate = _replace(parent_predicate, f"{other} = {alias}")
@ -222,7 +230,7 @@ def decorrelate(select, parent_select, external_columns, next_alias_name):
f"ARRAY_ANY({alias}, _x -> _x = {parent_predicate.this})",
)
else:
if is_subquery_projection:
if is_subquery_projection and select.parent.alias:
alias = exp.alias_(alias, select.parent.alias)
# COUNT always returns 0 on empty datasets, so we need take that into consideration here
@ -236,10 +244,7 @@ def decorrelate(select, parent_select, external_columns, next_alias_name):
return exp.null()
return node
alias = exp.Coalesce(
this=alias,
expressions=[value.this.transform(remove_aggs)],
)
alias = exp.Coalesce(this=alias, expressions=[value.this.transform(remove_aggs)])
select.parent.replace(alias)
@ -249,6 +254,8 @@ def decorrelate(select, parent_select, external_columns, next_alias_name):
if is_subquery_projection:
key.replace(nested)
if not isinstance(predicate, exp.EQ):
parent_select.where(predicate, copy=False)
continue
if key in group_by:

314
sqlglot/parser.py
View file

@ -61,6 +61,23 @@ def build_logarithm(args: t.List, dialect: Dialect) -> exp.Func:
return (exp.Ln if dialect.parser_class.LOG_DEFAULTS_TO_LN else exp.Log)(this=this)
def build_hex(args: t.List, dialect: Dialect) -> exp.Hex | exp.LowerHex:
arg = seq_get(args, 0)
return exp.LowerHex(this=arg) if dialect.HEX_LOWERCASE else exp.Hex(this=arg)
def build_lower(args: t.List) -> exp.Lower | exp.Hex:
# LOWER(HEX(..)) can be simplified to LowerHex to simplify its transpilation
arg = seq_get(args, 0)
return exp.LowerHex(this=arg.this) if isinstance(arg, exp.Hex) else exp.Lower(this=arg)
def build_upper(args: t.List) -> exp.Upper | exp.Hex:
# UPPER(HEX(..)) can be simplified to Hex to simplify its transpilation
arg = seq_get(args, 0)
return exp.Hex(this=arg.this) if isinstance(arg, exp.Hex) else exp.Upper(this=arg)
def build_extract_json_with_path(expr_type: t.Type[E]) -> t.Callable[[t.List, Dialect], E]:
def _builder(args: t.List, dialect: Dialect) -> E:
expression = expr_type(
@ -74,6 +91,17 @@ def build_extract_json_with_path(expr_type: t.Type[E]) -> t.Callable[[t.List, Di
return _builder
def build_mod(args: t.List) -> exp.Mod:
this = seq_get(args, 0)
expression = seq_get(args, 1)
# Wrap the operands if they are binary nodes, e.g. MOD(a + 1, 7) -> (a + 1) % 7
this = exp.Paren(this=this) if isinstance(this, exp.Binary) else this
expression = exp.Paren(this=expression) if isinstance(expression, exp.Binary) else expression
return exp.Mod(this=this, expression=expression)
class _Parser(type):
def __new__(cls, clsname, bases, attrs):
klass = super().__new__(cls, clsname, bases, attrs)
@ -123,7 +151,7 @@ class Parser(metaclass=_Parser):
"LOG": build_logarithm,
"LOG2": lambda args: exp.Log(this=exp.Literal.number(2), expression=seq_get(args, 0)),
"LOG10": lambda args: exp.Log(this=exp.Literal.number(10), expression=seq_get(args, 0)),
"MOD": lambda args: exp.Mod(this=seq_get(args, 0), expression=seq_get(args, 1)),
"MOD": build_mod,
"TIME_TO_TIME_STR": lambda args: exp.Cast(
this=seq_get(args, 0),
to=exp.DataType(this=exp.DataType.Type.TEXT),
@ -137,6 +165,10 @@ class Parser(metaclass=_Parser):
length=exp.Literal.number(10),
),
"VAR_MAP": build_var_map,
"LOWER": build_lower,
"UPPER": build_upper,
"HEX": build_hex,
"TO_HEX": build_hex,
}
NO_PAREN_FUNCTIONS = {
@ -295,22 +327,23 @@ class Parser(metaclass=_Parser):
DB_CREATABLES = {
TokenType.DATABASE,
TokenType.SCHEMA,
TokenType.TABLE,
TokenType.VIEW,
TokenType.MODEL,
TokenType.DICTIONARY,
TokenType.MODEL,
TokenType.SCHEMA,
TokenType.SEQUENCE,
TokenType.STORAGE_INTEGRATION,
TokenType.TABLE,
TokenType.TAG,
TokenType.VIEW,
}
CREATABLES = {
TokenType.COLUMN,
TokenType.CONSTRAINT,
TokenType.FOREIGN_KEY,
TokenType.FUNCTION,
TokenType.INDEX,
TokenType.PROCEDURE,
TokenType.FOREIGN_KEY,
*DB_CREATABLES,
}
@ -373,6 +406,7 @@ class Parser(metaclass=_Parser):
TokenType.REFRESH,
TokenType.REPLACE,
TokenType.RIGHT,
TokenType.ROLLUP,
TokenType.ROW,
TokenType.ROWS,
TokenType.SEMI,
@ -467,7 +501,6 @@ class Parser(metaclass=_Parser):
}
EQUALITY = {
TokenType.COLON_EQ: exp.PropertyEQ,
TokenType.EQ: exp.EQ,
TokenType.NEQ: exp.NEQ,
TokenType.NULLSAFE_EQ: exp.NullSafeEQ,
@ -653,6 +686,7 @@ class Parser(metaclass=_Parser):
kind=self._parse_var_from_options(self.USABLES, raise_unmatched=False),
this=self._parse_table(schema=False),
),
TokenType.SEMICOLON: lambda self: self.expression(exp.Semicolon),
}
UNARY_PARSERS = {
@ -700,7 +734,12 @@ class Parser(metaclass=_Parser):
TokenType.FALSE: lambda self, _: self.expression(exp.Boolean, this=False),
TokenType.SESSION_PARAMETER: lambda self, _: self._parse_session_parameter(),
TokenType.STAR: lambda self, _: self.expression(
exp.Star, **{"except": self._parse_except(), "replace": self._parse_replace()}
exp.Star,
**{
"except": self._parse_star_op("EXCEPT", "EXCLUDE"),
"replace": self._parse_star_op("REPLACE"),
"rename": self._parse_star_op("RENAME"),
},
),
}
@ -729,6 +768,9 @@ class Parser(metaclass=_Parser):
}
PROPERTY_PARSERS: t.Dict[str, t.Callable] = {
"ALLOWED_VALUES": lambda self: self.expression(
exp.AllowedValuesProperty, expressions=self._parse_csv(self._parse_primary)
),
"ALGORITHM": lambda self: self._parse_property_assignment(exp.AlgorithmProperty),
"AUTO": lambda self: self._parse_auto_property(),
"AUTO_INCREMENT": lambda self: self._parse_property_assignment(exp.AutoIncrementProperty),
@ -748,6 +790,7 @@ class Parser(metaclass=_Parser):
"CONTAINS": lambda self: self._parse_contains_property(),
"COPY": lambda self: self._parse_copy_property(),
"DATABLOCKSIZE": lambda self, **kwargs: self._parse_datablocksize(**kwargs),
"DATA_DELETION": lambda self: self._parse_data_deletion_property(),
"DEFINER": lambda self: self._parse_definer(),
"DETERMINISTIC": lambda self: self.expression(
exp.StabilityProperty, this=exp.Literal.string("IMMUTABLE")
@ -796,6 +839,7 @@ class Parser(metaclass=_Parser):
"READS": lambda self: self._parse_reads_property(),
"REMOTE": lambda self: self._parse_remote_with_connection(),
"RETURNS": lambda self: self._parse_returns(),
"STRICT": lambda self: self.expression(exp.StrictProperty),
"ROW": lambda self: self._parse_row(),
"ROW_FORMAT": lambda self: self._parse_property_assignment(exp.RowFormatProperty),
"SAMPLE": lambda self: self.expression(
@ -900,6 +944,14 @@ class Parser(metaclass=_Parser):
"DELETE": lambda self: self.expression(exp.Delete, where=self._parse_where()),
"DROP": lambda self: self._parse_alter_table_drop(),
"RENAME": lambda self: self._parse_alter_table_rename(),
"SET": lambda self: self._parse_alter_table_set(),
}
ALTER_ALTER_PARSERS = {
"DISTKEY": lambda self: self._parse_alter_diststyle(),
"DISTSTYLE": lambda self: self._parse_alter_diststyle(),
"SORTKEY": lambda self: self._parse_alter_sortkey(),
"COMPOUND": lambda self: self._parse_alter_sortkey(compound=True),
}
SCHEMA_UNNAMED_CONSTRAINTS = {
@ -990,6 +1042,8 @@ class Parser(metaclass=_Parser):
exp.DataType.Type.JSON: lambda self, this, _: self.expression(exp.ParseJSON, this=this),
}
TYPE_CONVERTER: t.Dict[exp.DataType.Type, t.Callable[[exp.DataType], exp.DataType]] = {}
DDL_SELECT_TOKENS = {TokenType.SELECT, TokenType.WITH, TokenType.L_PAREN}
PRE_VOLATILE_TOKENS = {TokenType.CREATE, TokenType.REPLACE, TokenType.UNIQUE}
@ -1084,6 +1138,9 @@ class Parser(metaclass=_Parser):
# Whether the table sample clause expects CSV syntax
TABLESAMPLE_CSV = False
# The default method used for table sampling
DEFAULT_SAMPLING_METHOD: t.Optional[str] = None
# Whether the SET command needs a delimiter (e.g. "=") for assignments
SET_REQUIRES_ASSIGNMENT_DELIMITER = True
@ -1228,6 +1285,9 @@ class Parser(metaclass=_Parser):
for i, token in enumerate(raw_tokens):
if token.token_type == TokenType.SEMICOLON:
if token.comments:
chunks.append([token])
if i < total - 1:
chunks.append([])
else:
@ -1471,7 +1531,7 @@ class Parser(metaclass=_Parser):
if self._match_set(self.STATEMENT_PARSERS):
return self.STATEMENT_PARSERS[self._prev.token_type](self)
if self._match_set(Tokenizer.COMMANDS):
if self._match_set(self.dialect.tokenizer.COMMANDS):
return self._parse_command()
expression = self._parse_expression()
@ -1492,6 +1552,8 @@ class Parser(metaclass=_Parser):
schema=True, is_db_reference=self._prev.token_type == TokenType.SCHEMA
)
cluster = self._parse_on_property() if self._match(TokenType.ON) else None
if self._match(TokenType.L_PAREN, advance=False):
expressions = self._parse_wrapped_csv(self._parse_types)
else:
@ -1503,12 +1565,13 @@ class Parser(metaclass=_Parser):
exists=if_exists,
this=table,
expressions=expressions,
kind=kind,
kind=kind.upper(),
temporary=temporary,
materialized=materialized,
cascade=self._match_text_seq("CASCADE"),
constraints=self._match_text_seq("CONSTRAINTS"),
purge=self._match_text_seq("PURGE"),
cluster=cluster,
)
def _parse_exists(self, not_: bool = False) -> t.Optional[bool]:
@ -1651,7 +1714,7 @@ class Parser(metaclass=_Parser):
exp.Clone, this=self._parse_table(schema=True), shallow=shallow, copy=copy
)
if self._curr:
if self._curr and not self._match_set((TokenType.R_PAREN, TokenType.COMMA), advance=False):
return self._parse_as_command(start)
return self.expression(
@ -1678,6 +1741,7 @@ class Parser(metaclass=_Parser):
index = self._index
while self._curr:
self._match(TokenType.COMMA)
if self._match_text_seq("INCREMENT"):
self._match_text_seq("BY")
self._match_text_seq("=")
@ -1822,23 +1886,65 @@ class Parser(metaclass=_Parser):
return self.expression(exp.StabilityProperty, this=exp.Literal.string("VOLATILE"))
def _parse_system_versioning_property(self) -> exp.WithSystemVersioningProperty:
self._match_pair(TokenType.EQ, TokenType.ON)
def _parse_retention_period(self) -> exp.Var:
# Parse TSQL's HISTORY_RETENTION_PERIOD: {INFINITE | <number> DAY | DAYS | MONTH ...}
number = self._parse_number()
number_str = f"{number} " if number else ""
unit = self._parse_var(any_token=True)
return exp.var(f"{number_str}{unit}")
prop = self.expression(exp.WithSystemVersioningProperty)
def _parse_system_versioning_property(
self, with_: bool = False
) -> exp.WithSystemVersioningProperty:
self._match(TokenType.EQ)
prop = self.expression(
exp.WithSystemVersioningProperty,
**{ # type: ignore
"on": True,
"with": with_,
},
)
if self._match_text_seq("OFF"):
prop.set("on", False)
return prop
self._match(TokenType.ON)
if self._match(TokenType.L_PAREN):
self._match_text_seq("HISTORY_TABLE", "=")
prop.set("this", self._parse_table_parts())
while self._curr and not self._match(TokenType.R_PAREN):
if self._match_text_seq("HISTORY_TABLE", "="):
prop.set("this", self._parse_table_parts())
elif self._match_text_seq("DATA_CONSISTENCY_CHECK", "="):
prop.set("data_consistency", self._advance_any() and self._prev.text.upper())
elif self._match_text_seq("HISTORY_RETENTION_PERIOD", "="):
prop.set("retention_period", self._parse_retention_period())
if self._match(TokenType.COMMA):
self._match_text_seq("DATA_CONSISTENCY_CHECK", "=")
prop.set("expression", self._advance_any() and self._prev.text.upper())
self._match(TokenType.COMMA)
self._match_r_paren()
return prop
def _parse_data_deletion_property(self) -> exp.DataDeletionProperty:
self._match(TokenType.EQ)
on = self._match_text_seq("ON") or not self._match_text_seq("OFF")
prop = self.expression(exp.DataDeletionProperty, on=on)
if self._match(TokenType.L_PAREN):
while self._curr and not self._match(TokenType.R_PAREN):
if self._match_text_seq("FILTER_COLUMN", "="):
prop.set("filter_column", self._parse_column())
elif self._match_text_seq("RETENTION_PERIOD", "="):
prop.set("retention_period", self._parse_retention_period())
self._match(TokenType.COMMA)
return prop
def _parse_with_property(self) -> t.Optional[exp.Expression] | t.List[exp.Expression]:
if self._match_text_seq("(", "SYSTEM_VERSIONING"):
prop = self._parse_system_versioning_property(with_=True)
self._match_r_paren()
return prop
if self._match(TokenType.L_PAREN, advance=False):
return self._parse_wrapped_properties()
@ -1853,6 +1959,9 @@ class Parser(metaclass=_Parser):
elif self._match_text_seq("NO", "DATA"):
return self._parse_withdata(no=True)
if self._match(TokenType.SERDE_PROPERTIES, advance=False):
return self._parse_serde_properties(with_=True)
if not self._next:
return None
@ -2201,6 +2310,7 @@ class Parser(metaclass=_Parser):
def _parse_returns(self) -> exp.ReturnsProperty:
value: t.Optional[exp.Expression]
null = None
is_table = self._match(TokenType.TABLE)
if is_table:
@ -2214,10 +2324,13 @@ class Parser(metaclass=_Parser):
self.raise_error("Expecting >")
else:
value = self._parse_schema(exp.var("TABLE"))
elif self._match_text_seq("NULL", "ON", "NULL", "INPUT"):
null = True
value = None
else:
value = self._parse_types()
return self.expression(exp.ReturnsProperty, this=value, is_table=is_table)
return self.expression(exp.ReturnsProperty, this=value, is_table=is_table, null=null)
def _parse_describe(self) -> exp.Describe:
kind = self._match_set(self.CREATABLES) and self._prev.text
@ -2340,6 +2453,21 @@ class Parser(metaclass=_Parser):
return None
return self._parse_row_format()
def _parse_serde_properties(self, with_: bool = False) -> t.Optional[exp.SerdeProperties]:
index = self._index
with_ = with_ or self._match_text_seq("WITH")
if not self._match(TokenType.SERDE_PROPERTIES):
self._retreat(index)
return None
return self.expression(
exp.SerdeProperties,
**{ # type: ignore
"expressions": self._parse_wrapped_properties(),
"with": with_,
},
)
def _parse_row_format(
self, match_row: bool = False
) -> t.Optional[exp.RowFormatSerdeProperty | exp.RowFormatDelimitedProperty]:
@ -2349,11 +2477,7 @@ class Parser(metaclass=_Parser):
if self._match_text_seq("SERDE"):
this = self._parse_string()
serde_properties = None
if self._match(TokenType.SERDE_PROPERTIES):
serde_properties = self.expression(
exp.SerdeProperties, expressions=self._parse_wrapped_properties()
)
serde_properties = self._parse_serde_properties()
return self.expression(
exp.RowFormatSerdeProperty, this=this, serde_properties=serde_properties
@ -2672,9 +2796,7 @@ class Parser(metaclass=_Parser):
)
def _implicit_unnests_to_explicit(self, this: E) -> E:
from sqlglot.optimizer.normalize_identifiers import (
normalize_identifiers as _norm,
)
from sqlglot.optimizer.normalize_identifiers import normalize_identifiers as _norm
refs = {_norm(this.args["from"].this.copy(), dialect=self.dialect).alias_or_name}
for i, join in enumerate(this.args.get("joins") or []):
@ -3366,6 +3488,9 @@ class Parser(metaclass=_Parser):
elif self._match_texts(("SEED", "REPEATABLE")):
seed = self._parse_wrapped(self._parse_number)
if not method and self.DEFAULT_SAMPLING_METHOD:
method = exp.var(self.DEFAULT_SAMPLING_METHOD)
return self.expression(
exp.TableSample,
expressions=expressions,
@ -3519,7 +3644,11 @@ class Parser(metaclass=_Parser):
elements["all"] = False
while True:
expressions = self._parse_csv(self._parse_conjunction)
expressions = self._parse_csv(
lambda: None
if self._match(TokenType.ROLLUP, advance=False)
else self._parse_conjunction()
)
if expressions:
elements["expressions"].extend(expressions)
@ -3817,7 +3946,24 @@ class Parser(metaclass=_Parser):
return self._parse_alias(self._parse_conjunction())
def _parse_conjunction(self) -> t.Optional[exp.Expression]:
return self._parse_tokens(self._parse_equality, self.CONJUNCTION)
this = self._parse_equality()
if self._match(TokenType.COLON_EQ):
this = self.expression(
exp.PropertyEQ,
this=this,
comments=self._prev_comments,
expression=self._parse_conjunction(),
)
while self._match_set(self.CONJUNCTION):
this = self.expression(
self.CONJUNCTION[self._prev.token_type],
this=this,
comments=self._prev_comments,
expression=self._parse_equality(),
)
return this
def _parse_equality(self) -> t.Optional[exp.Expression]:
return self._parse_tokens(self._parse_comparison, self.EQUALITY)
@ -4061,6 +4207,7 @@ class Parser(metaclass=_Parser):
) -> t.Optional[exp.Expression]:
index = self._index
this: t.Optional[exp.Expression] = None
prefix = self._match_text_seq("SYSUDTLIB", ".")
if not self._match_set(self.TYPE_TOKENS):
@ -4081,7 +4228,7 @@ class Parser(metaclass=_Parser):
while self._match(TokenType.DOT):
type_name = f"{type_name}.{self._advance_any() and self._prev.text}"
return exp.DataType.build(type_name, udt=True)
this = exp.DataType.build(type_name, udt=True)
else:
self._retreat(self._index - 1)
return None
@ -4134,7 +4281,6 @@ class Parser(metaclass=_Parser):
maybe_func = True
this: t.Optional[exp.Expression] = None
values: t.Optional[t.List[exp.Expression]] = None
if nested and self._match(TokenType.LT):
@ -4203,10 +4349,17 @@ class Parser(metaclass=_Parser):
values=values,
prefix=prefix,
)
elif expressions:
this.set("expressions", expressions)
while self._match_pair(TokenType.L_BRACKET, TokenType.R_BRACKET):
this = exp.DataType(this=exp.DataType.Type.ARRAY, expressions=[this], nested=True)
if self.TYPE_CONVERTER and isinstance(this.this, exp.DataType.Type):
converter = self.TYPE_CONVERTER.get(this.this)
if converter:
this = converter(t.cast(exp.DataType, this))
return this
def _parse_struct_types(self, type_required: bool = False) -> t.Optional[exp.Expression]:
@ -4326,9 +4479,7 @@ class Parser(metaclass=_Parser):
if not this and self._match(TokenType.R_PAREN, advance=False):
this = self.expression(exp.Tuple)
elif isinstance(this, exp.UNWRAPPED_QUERIES):
this = self._parse_set_operations(
self._parse_subquery(this=this, parse_alias=False)
)
this = self._parse_subquery(this=this, parse_alias=False)
elif isinstance(this, exp.Subquery):
this = self._parse_subquery(
this=self._parse_set_operations(this), parse_alias=False
@ -5625,13 +5776,8 @@ class Parser(metaclass=_Parser):
return self._parse_placeholder()
def _parse_parameter(self) -> exp.Parameter:
self._match(TokenType.L_BRACE)
this = self._parse_identifier() or self._parse_primary_or_var()
expression = self._match(TokenType.COLON) and (
self._parse_identifier() or self._parse_primary_or_var()
)
self._match(TokenType.R_BRACE)
return self.expression(exp.Parameter, this=this, expression=expression)
return self.expression(exp.Parameter, this=this)
def _parse_placeholder(self) -> t.Optional[exp.Expression]:
if self._match_set(self.PLACEHOLDER_PARSERS):
@ -5641,23 +5787,14 @@ class Parser(metaclass=_Parser):
self._advance(-1)
return None
def _parse_except(self) -> t.Optional[t.List[exp.Expression]]:
if not self._match(TokenType.EXCEPT):
return None
if self._match(TokenType.L_PAREN, advance=False):
return self._parse_wrapped_csv(self._parse_column)
except_column = self._parse_column()
return [except_column] if except_column else None
def _parse_replace(self) -> t.Optional[t.List[exp.Expression]]:
if not self._match(TokenType.REPLACE):
def _parse_star_op(self, *keywords: str) -> t.Optional[t.List[exp.Expression]]:
if not self._match_texts(keywords):
return None
if self._match(TokenType.L_PAREN, advance=False):
return self._parse_wrapped_csv(self._parse_expression)
replace_expression = self._parse_expression()
return [replace_expression] if replace_expression else None
expression = self._parse_expression()
return [expression] if expression else None
def _parse_csv(
self, parse_method: t.Callable, sep: TokenType = TokenType.COMMA
@ -5812,7 +5949,12 @@ class Parser(metaclass=_Parser):
return self._parse_wrapped_csv(self._parse_field_def, optional=True)
return self._parse_wrapped_csv(self._parse_add_column, optional=True)
def _parse_alter_table_alter(self) -> exp.AlterColumn:
def _parse_alter_table_alter(self) -> t.Optional[exp.Expression]:
if self._match_texts(self.ALTER_ALTER_PARSERS):
return self.ALTER_ALTER_PARSERS[self._prev.text.upper()](self)
# Many dialects support the ALTER [COLUMN] syntax, so if there is no
# keyword after ALTER we default to parsing this statement
self._match(TokenType.COLUMN)
column = self._parse_field(any_token=True)
@ -5833,6 +5975,27 @@ class Parser(metaclass=_Parser):
using=self._match(TokenType.USING) and self._parse_conjunction(),
)
def _parse_alter_diststyle(self) -> exp.AlterDistStyle:
if self._match_texts(("ALL", "EVEN", "AUTO")):
return self.expression(exp.AlterDistStyle, this=exp.var(self._prev.text.upper()))
self._match_text_seq("KEY", "DISTKEY")
return self.expression(exp.AlterDistStyle, this=self._parse_column())
def _parse_alter_sortkey(self, compound: t.Optional[bool] = None) -> exp.AlterSortKey:
if compound:
self._match_text_seq("SORTKEY")
if self._match(TokenType.L_PAREN, advance=False):
return self.expression(
exp.AlterSortKey, expressions=self._parse_wrapped_id_vars(), compound=compound
)
self._match_texts(("AUTO", "NONE"))
return self.expression(
exp.AlterSortKey, this=exp.var(self._prev.text.upper()), compound=compound
)
def _parse_alter_table_drop(self) -> t.List[exp.Expression]:
index = self._index - 1
@ -5858,6 +6021,41 @@ class Parser(metaclass=_Parser):
self._match_text_seq("TO")
return self.expression(exp.RenameTable, this=self._parse_table(schema=True))
def _parse_alter_table_set(self) -> exp.AlterSet:
alter_set = self.expression(exp.AlterSet)
if self._match(TokenType.L_PAREN, advance=False) or self._match_text_seq(
"TABLE", "PROPERTIES"
):
alter_set.set("expressions", self._parse_wrapped_csv(self._parse_conjunction))
elif self._match_text_seq("FILESTREAM_ON", advance=False):
alter_set.set("expressions", [self._parse_conjunction()])
elif self._match_texts(("LOGGED", "UNLOGGED")):
alter_set.set("option", exp.var(self._prev.text.upper()))
elif self._match_text_seq("WITHOUT") and self._match_texts(("CLUSTER", "OIDS")):
alter_set.set("option", exp.var(f"WITHOUT {self._prev.text.upper()}"))
elif self._match_text_seq("LOCATION"):
alter_set.set("location", self._parse_field())
elif self._match_text_seq("ACCESS", "METHOD"):
alter_set.set("access_method", self._parse_field())
elif self._match_text_seq("TABLESPACE"):
alter_set.set("tablespace", self._parse_field())
elif self._match_text_seq("FILE", "FORMAT") or self._match_text_seq("FILEFORMAT"):
alter_set.set("file_format", [self._parse_field()])
elif self._match_text_seq("STAGE_FILE_FORMAT"):
alter_set.set("file_format", self._parse_wrapped_options())
elif self._match_text_seq("STAGE_COPY_OPTIONS"):
alter_set.set("copy_options", self._parse_wrapped_options())
elif self._match_text_seq("TAG") or self._match_text_seq("TAGS"):
alter_set.set("tag", self._parse_csv(self._parse_conjunction))
else:
if self._match_text_seq("SERDE"):
alter_set.set("serde", self._parse_field())
alter_set.set("expressions", [self._parse_properties()])
return alter_set
def _parse_alter(self) -> exp.AlterTable | exp.Command:
start = self._prev
@ -5867,6 +6065,7 @@ class Parser(metaclass=_Parser):
exists = self._parse_exists()
only = self._match_text_seq("ONLY")
this = self._parse_table(schema=True)
cluster = self._parse_on_property() if self._match(TokenType.ON) else None
if self._next:
self._advance()
@ -5884,6 +6083,7 @@ class Parser(metaclass=_Parser):
actions=actions,
only=only,
options=options,
cluster=cluster,
)
return self._parse_as_command(start)
@ -5974,7 +6174,7 @@ class Parser(metaclass=_Parser):
if kind in ("GLOBAL", "SESSION") and self._match_text_seq("TRANSACTION"):
return self._parse_set_transaction(global_=kind == "GLOBAL")
left = self._parse_primary() or self._parse_id_var()
left = self._parse_primary() or self._parse_column()
assignment_delimiter = self._match_texts(("=", "TO"))
if not left or (self.SET_REQUIRES_ASSIGNMENT_DELIMITER and not assignment_delimiter):

22
sqlglot/schema.py
View file

@ -6,7 +6,7 @@ import typing as t
from sqlglot import expressions as exp
from sqlglot.dialects.dialect import Dialect
from sqlglot.errors import SchemaError
from sqlglot.helper import dict_depth
from sqlglot.helper import dict_depth, first
from sqlglot.trie import TrieResult, in_trie, new_trie
if t.TYPE_CHECKING:
@ -174,7 +174,7 @@ class AbstractMappingSchema:
return None
if value == TrieResult.PREFIX:
possibilities = flatten_schema(trie, depth=dict_depth(trie) - 1)
possibilities = flatten_schema(trie)
if len(possibilities) == 1:
parts.extend(possibilities[0])
@ -362,14 +362,19 @@ class MappingSchema(AbstractMappingSchema, Schema):
The normalized schema mapping.
"""
normalized_mapping: t.Dict = {}
flattened_schema = flatten_schema(schema, depth=dict_depth(schema) - 1)
flattened_schema = flatten_schema(schema)
error_msg = "Table {} must match the schema's nesting level: {}."
for keys in flattened_schema:
columns = nested_get(schema, *zip(keys, keys))
if not isinstance(columns, dict):
raise SchemaError(error_msg.format(".".join(keys[:-1]), len(flattened_schema[0])))
if isinstance(first(columns.values()), dict):
raise SchemaError(
f"Table {'.'.join(keys[:-1])} must match the schema's nesting level: {len(flattened_schema[0])}."
error_msg.format(
".".join(keys + flatten_schema(columns)[0]), len(flattened_schema[0])
),
)
normalized_keys = [self._normalize_name(key, is_table=True) for key in keys]
@ -494,16 +499,17 @@ def ensure_column_mapping(mapping: t.Optional[ColumnMapping]) -> t.Dict:
def flatten_schema(
schema: t.Dict, depth: int, keys: t.Optional[t.List[str]] = None
schema: t.Dict, depth: t.Optional[int] = None, keys: t.Optional[t.List[str]] = None
) -> t.List[t.List[str]]:
tables = []
keys = keys or []
depth = dict_depth(schema) - 1 if depth is None else depth
for k, v in schema.items():
if depth >= 2:
tables.extend(flatten_schema(v, depth - 1, keys + [k]))
elif depth == 1:
if depth == 1 or not isinstance(v, dict):
tables.append(keys + [k])
elif depth >= 2:
tables.extend(flatten_schema(v, depth - 1, keys + [k]))
return tables

4
sqlglot/tokens.py
View file

@ -234,6 +234,7 @@ class TokenType(AutoName):
CURRENT_TIME = auto()
CURRENT_TIMESTAMP = auto()
CURRENT_USER = auto()
DECLARE = auto()
DEFAULT = auto()
DELETE = auto()
DESC = auto()
@ -358,6 +359,7 @@ class TokenType(AutoName):
STORAGE_INTEGRATION = auto()
STRUCT = auto()
TABLE_SAMPLE = auto()
TAG = auto()
TEMPORARY = auto()
TOP = auto()
THEN = auto()
@ -1187,6 +1189,8 @@ class Tokenizer(metaclass=_Tokenizer):
if self._peek.isdigit():
self._advance()
elif self._peek == "." and not decimal:
if self.tokens and self.tokens[-1].token_type == TokenType.PARAMETER:
return self._add(TokenType.NUMBER)
decimal = True
self._advance()
elif self._peek in ("-", "+") and scientific == 1:

28
sqlglot/transforms.py
View file

@ -105,7 +105,14 @@ def eliminate_qualify(expression: exp.Expression) -> exp.Expression:
select.replace(exp.alias_(select, alias))
taken.add(alias)
outer_selects = exp.select(*[select.alias_or_name for select in expression.selects])
def _select_alias_or_name(select: exp.Expression) -> str | exp.Column:
alias_or_name = select.alias_or_name
identifier = select.args.get("alias") or select.this
if isinstance(identifier, exp.Identifier):
return exp.column(alias_or_name, quoted=identifier.args.get("quoted"))
return alias_or_name
outer_selects = exp.select(*list(map(_select_alias_or_name, expression.selects)))
qualify_filters = expression.args["qualify"].pop().this
expression_by_alias = {
select.alias: select.this
@ -465,19 +472,28 @@ def move_ctes_to_top_level(expression: exp.Expression) -> exp.Expression:
TODO: handle name clashes whilst moving CTEs (it can get quite tricky & costly).
"""
top_level_with = expression.args.get("with")
for node in expression.find_all(exp.With):
if node.parent is expression:
for inner_with in expression.find_all(exp.With):
if inner_with.parent is expression:
continue
inner_with = node.pop()
if not top_level_with:
top_level_with = inner_with
top_level_with = inner_with.pop()
expression.set("with", top_level_with)
else:
if inner_with.recursive:
top_level_with.set("recursive", True)
top_level_with.set("expressions", inner_with.expressions + top_level_with.expressions)
parent_cte = inner_with.find_ancestor(exp.CTE)
inner_with.pop()
if parent_cte:
i = top_level_with.expressions.index(parent_cte)
top_level_with.expressions[i:i] = inner_with.expressions
top_level_with.set("expressions", top_level_with.expressions)
else:
top_level_with.set(
"expressions", top_level_with.expressions + inner_with.expressions
)
return expression

2
sqlglotrs/Cargo.lock generated
View file

@ -188,7 +188,7 @@ checksum = "4dccd0940a2dcdf68d092b8cbab7dc0ad8fa938bf95787e1b916b0e3d0e8e970"
[[package]]
name = "sqlglotrs"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"pyo3",
]

2
sqlglotrs/Cargo.toml
View file

@ -1,6 +1,6 @@
[package]
name = "sqlglotrs"
version = "0.2.0"
version = "0.2.5"
edition = "2021"
[lib]

3
sqlglotrs/src/tokenizer.rs
View file

@ -483,6 +483,9 @@ impl<'a> TokenizerState<'a> {
if self.peek_char.is_digit(10) {
self.advance(1)?;
} else if self.peek_char == '.' && !decimal {
if self.tokens.last().map(|t| t.token_type) == Some(self.token_types.parameter) {
return self.add(self.token_types.number, None);
}
decimal = true;
self.advance(1)?;
} else if (self.peek_char == '-' || self.peek_char == '+') && scientific == 1 {

217
tests/dialects/test_bigquery.py
View file

@ -20,6 +20,13 @@ class TestBigQuery(Validator):
maxDiff = None
def test_bigquery(self):
self.validate_identity(
"""CREATE TEMPORARY FUNCTION FOO()
RETURNS STRING
LANGUAGE js AS
'return "Hello world!"'""",
pretty=True,
)
self.validate_identity(
"[a, a(1, 2,3,4444444444444444, tttttaoeunthaoentuhaoentuheoantu, toheuntaoheutnahoeunteoahuntaoeh), b(3, 4,5), c, d, tttttttttttttttteeeeeeeeeeeeeett, 12312312312]",
"""[
@ -278,6 +285,13 @@ class TestBigQuery(Validator):
r"REGEXP_EXTRACT(svc_plugin_output, '\\\\\\((.*)')",
)
self.validate_all(
"SELECT t.c1, h.c2, s.c3 FROM t1 AS t, UNNEST(t.t2) AS h, UNNEST(h.t3) AS s",
write={
"bigquery": "SELECT t.c1, h.c2, s.c3 FROM t1 AS t, UNNEST(t.t2) AS h, UNNEST(h.t3) AS s",
"duckdb": "SELECT t.c1, h.c2, s.c3 FROM t1 AS t, UNNEST(t.t2) AS _t0(h), UNNEST(h.t3) AS _t1(s)",
},
)
self.validate_all(
"PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%E6S%z', x)",
write={
@ -289,7 +303,7 @@ class TestBigQuery(Validator):
"SELECT results FROM Coordinates, Coordinates.position AS results",
write={
"bigquery": "SELECT results FROM Coordinates, UNNEST(Coordinates.position) AS results",
"presto": "SELECT results FROM Coordinates, UNNEST(Coordinates.position) AS _t(results)",
"presto": "SELECT results FROM Coordinates, UNNEST(Coordinates.position) AS _t0(results)",
},
)
self.validate_all(
@ -307,7 +321,7 @@ class TestBigQuery(Validator):
},
write={
"bigquery": "SELECT results FROM Coordinates AS c, UNNEST(c.position) AS results",
"presto": "SELECT results FROM Coordinates AS c, UNNEST(c.position) AS _t(results)",
"presto": "SELECT results FROM Coordinates AS c, UNNEST(c.position) AS _t0(results)",
"redshift": "SELECT results FROM Coordinates AS c, c.position AS results",
},
)
@ -525,7 +539,7 @@ class TestBigQuery(Validator):
"SELECT * FROM t WHERE EXISTS(SELECT * FROM unnest(nums) AS x WHERE x > 1)",
write={
"bigquery": "SELECT * FROM t WHERE EXISTS(SELECT * FROM UNNEST(nums) AS x WHERE x > 1)",
"duckdb": "SELECT * FROM t WHERE EXISTS(SELECT * FROM UNNEST(nums) AS _t(x) WHERE x > 1)",
"duckdb": "SELECT * FROM t WHERE EXISTS(SELECT * FROM UNNEST(nums) AS _t0(x) WHERE x > 1)",
},
)
self.validate_all(
@ -617,13 +631,88 @@ class TestBigQuery(Validator):
"mysql": "SELECT DATE_SUB(TIMESTAMP('2008-12-25 15:30:00+00'), INTERVAL 10 MINUTE)",
},
)
self.validate_all(
"LOWER(TO_HEX(x))",
write={
"": "LOWER(HEX(x))",
"bigquery": "TO_HEX(x)",
"clickhouse": "LOWER(HEX(x))",
"duckdb": "LOWER(HEX(x))",
"hive": "LOWER(HEX(x))",
"mysql": "LOWER(HEX(x))",
"spark": "LOWER(HEX(x))",
"sqlite": "LOWER(HEX(x))",
"presto": "LOWER(TO_HEX(x))",
"trino": "LOWER(TO_HEX(x))",
},
)
self.validate_all(
"TO_HEX(x)",
read={
"": "LOWER(HEX(x))",
"clickhouse": "LOWER(HEX(x))",
"duckdb": "LOWER(HEX(x))",
"hive": "LOWER(HEX(x))",
"mysql": "LOWER(HEX(x))",
"spark": "LOWER(HEX(x))",
"sqlite": "LOWER(HEX(x))",
"presto": "LOWER(TO_HEX(x))",
"trino": "LOWER(TO_HEX(x))",
},
write={
"": "LOWER(HEX(x))",
"bigquery": "TO_HEX(x)",
"clickhouse": "LOWER(HEX(x))",
"duckdb": "LOWER(HEX(x))",
"hive": "LOWER(HEX(x))",
"mysql": "LOWER(HEX(x))",
"presto": "LOWER(TO_HEX(x))",
"spark": "LOWER(HEX(x))",
"sqlite": "LOWER(HEX(x))",
"trino": "LOWER(TO_HEX(x))",
},
)
self.validate_all(
"UPPER(TO_HEX(x))",
read={
"": "HEX(x)",
"clickhouse": "HEX(x)",
"duckdb": "HEX(x)",
"hive": "HEX(x)",
"mysql": "HEX(x)",
"presto": "TO_HEX(x)",
"spark": "HEX(x)",
"sqlite": "HEX(x)",
"trino": "TO_HEX(x)",
},
write={
"": "HEX(x)",
"bigquery": "UPPER(TO_HEX(x))",
"clickhouse": "HEX(x)",
"duckdb": "HEX(x)",
"hive": "HEX(x)",
"mysql": "HEX(x)",
"presto": "TO_HEX(x)",
"spark": "HEX(x)",
"sqlite": "HEX(x)",
"trino": "TO_HEX(x)",
},
)
self.validate_all(
"MD5(x)",
read={
"clickhouse": "MD5(x)",
"presto": "MD5(x)",
"trino": "MD5(x)",
},
write={
"": "MD5_DIGEST(x)",
"bigquery": "MD5(x)",
"clickhouse": "MD5(x)",
"hive": "UNHEX(MD5(x))",
"presto": "MD5(x)",
"spark": "UNHEX(MD5(x))",
"trino": "MD5(x)",
},
)
self.validate_all(
@ -631,25 +720,69 @@ class TestBigQuery(Validator):
read={
"duckdb": "SELECT MD5(some_string)",
"spark": "SELECT MD5(some_string)",
"clickhouse": "SELECT LOWER(HEX(MD5(some_string)))",
"presto": "SELECT LOWER(TO_HEX(MD5(some_string)))",
"trino": "SELECT LOWER(TO_HEX(MD5(some_string)))",
},
write={
"": "SELECT MD5(some_string)",
"bigquery": "SELECT TO_HEX(MD5(some_string))",
"duckdb": "SELECT MD5(some_string)",
"clickhouse": "SELECT LOWER(HEX(MD5(some_string)))",
"presto": "SELECT LOWER(TO_HEX(MD5(some_string)))",
"trino": "SELECT LOWER(TO_HEX(MD5(some_string)))",
},
)
self.validate_all(
"SHA1(x)",
read={
"clickhouse": "SHA1(x)",
"presto": "SHA1(x)",
"trino": "SHA1(x)",
},
write={
"clickhouse": "SHA1(x)",
"bigquery": "SHA1(x)",
"": "SHA(x)",
"presto": "SHA1(x)",
"trino": "SHA1(x)",
},
)
self.validate_all(
"SHA1(x)",
write={
"bigquery": "SHA1(x)",
"": "SHA(x)",
},
)
self.validate_all(
"SHA256(x)",
read={
"clickhouse": "SHA256(x)",
"presto": "SHA256(x)",
"trino": "SHA256(x)",
},
write={
"bigquery": "SHA256(x)",
"spark2": "SHA2(x, 256)",
"clickhouse": "SHA256(x)",
"presto": "SHA256(x)",
"trino": "SHA256(x)",
},
)
self.validate_all(
"SHA512(x)",
read={
"clickhouse": "SHA512(x)",
"presto": "SHA512(x)",
"trino": "SHA512(x)",
},
write={
"clickhouse": "SHA512(x)",
"bigquery": "SHA512(x)",
"spark2": "SHA2(x, 512)",
"presto": "SHA512(x)",
"trino": "SHA512(x)",
},
)
self.validate_all(
@ -860,8 +993,8 @@ class TestBigQuery(Validator):
},
write={
"bigquery": "SELECT * FROM UNNEST(['7', '14']) AS x",
"presto": "SELECT * FROM UNNEST(ARRAY['7', '14']) AS _t(x)",
"spark": "SELECT * FROM UNNEST(ARRAY('7', '14')) AS _t(x)",
"presto": "SELECT * FROM UNNEST(ARRAY['7', '14']) AS _t0(x)",
"spark": "SELECT * FROM UNNEST(ARRAY('7', '14')) AS _t0(x)",
},
)
self.validate_all(
@ -981,6 +1114,69 @@ class TestBigQuery(Validator):
"bigquery": "CREATE TABLE db.example_table (x INT64) PARTITION BY x CLUSTER BY x",
},
)
self.validate_all(
"DELETE db.example_table WHERE x = 1",
write={
"bigquery": "DELETE db.example_table WHERE x = 1",
"presto": "DELETE FROM db.example_table WHERE x = 1",
},
)
self.validate_all(
"DELETE db.example_table tb WHERE tb.x = 1",
write={
"bigquery": "DELETE db.example_table AS tb WHERE tb.x = 1",
"presto": "DELETE FROM db.example_table WHERE x = 1",
},
)
self.validate_all(
"DELETE db.example_table AS tb WHERE tb.x = 1",
write={
"bigquery": "DELETE db.example_table AS tb WHERE tb.x = 1",
"presto": "DELETE FROM db.example_table WHERE x = 1",
},
)
self.validate_all(
"DELETE FROM db.example_table WHERE x = 1",
write={
"bigquery": "DELETE FROM db.example_table WHERE x = 1",
"presto": "DELETE FROM db.example_table WHERE x = 1",
},
)
self.validate_all(
"DELETE FROM db.example_table tb WHERE tb.x = 1",
write={
"bigquery": "DELETE FROM db.example_table AS tb WHERE tb.x = 1",
"presto": "DELETE FROM db.example_table WHERE x = 1",
},
)
self.validate_all(
"DELETE FROM db.example_table AS tb WHERE tb.x = 1",
write={
"bigquery": "DELETE FROM db.example_table AS tb WHERE tb.x = 1",
"presto": "DELETE FROM db.example_table WHERE x = 1",
},
)
self.validate_all(
"DELETE FROM db.example_table AS tb WHERE example_table.x = 1",
write={
"bigquery": "DELETE FROM db.example_table AS tb WHERE example_table.x = 1",
"presto": "DELETE FROM db.example_table WHERE x = 1",
},
)
self.validate_all(
"DELETE FROM db.example_table WHERE example_table.x = 1",
write={
"bigquery": "DELETE FROM db.example_table WHERE example_table.x = 1",
"presto": "DELETE FROM db.example_table WHERE example_table.x = 1",
},
)
self.validate_all(
"DELETE FROM db.t1 AS t1 WHERE NOT t1.c IN (SELECT db.t2.c FROM db.t2)",
write={
"bigquery": "DELETE FROM db.t1 AS t1 WHERE NOT t1.c IN (SELECT db.t2.c FROM db.t2)",
"presto": "DELETE FROM db.t1 WHERE NOT c IN (SELECT c FROM db.t2)",
},
)
self.validate_all(
"SELECT * FROM a WHERE b IN UNNEST([1, 2, 3])",
write={
@ -1464,3 +1660,14 @@ OPTIONS (
with self.assertRaises(ParseError):
transpile("SELECT JSON_OBJECT('a', 1, 'b') AS json_data", read="bigquery")
def test_mod(self):
for sql in ("MOD(a, b)", "MOD('a', b)", "MOD(5, 2)", "MOD((a + 1) * 8, 5 - 1)"):
with self.subTest(f"Testing BigQuery roundtrip of modulo operation: {sql}"):
self.validate_identity(sql)
self.validate_identity("SELECT MOD((SELECT 1), 2)")
self.validate_identity(
"MOD((a + 1), b)",
"MOD(a + 1, b)",
)

23
tests/dialects/test_clickhouse.py
View file

@ -425,6 +425,21 @@ class TestClickhouse(Validator):
},
)
self.validate_identity("ALTER TABLE visits DROP PARTITION 201901")
self.validate_identity("ALTER TABLE visits DROP PARTITION ALL")
self.validate_identity(
"ALTER TABLE visits DROP PARTITION tuple(toYYYYMM(toDate('2019-01-25')))"
)
self.validate_identity("ALTER TABLE visits DROP PARTITION ID '201901'")
self.validate_identity("ALTER TABLE visits REPLACE PARTITION 201901 FROM visits_tmp")
self.validate_identity("ALTER TABLE visits REPLACE PARTITION ALL FROM visits_tmp")
self.validate_identity(
"ALTER TABLE visits REPLACE PARTITION tuple(toYYYYMM(toDate('2019-01-25'))) FROM visits_tmp"
)
self.validate_identity("ALTER TABLE visits REPLACE PARTITION ID '201901' FROM visits_tmp")
self.validate_identity("ALTER TABLE visits ON CLUSTER test_cluster DROP COLUMN col1")
def test_cte(self):
self.validate_identity("WITH 'x' AS foo SELECT foo")
self.validate_identity("WITH ['c'] AS field_names SELECT field_names")
@ -829,6 +844,9 @@ LIFETIME(MIN 0 MAX 0)""",
self.validate_identity(
"CREATE TABLE t1 (a String EPHEMERAL, b String EPHEMERAL func(), c String MATERIALIZED func(), d String ALIAS func()) ENGINE=TinyLog()"
)
self.validate_identity(
"CREATE TABLE t (a String, b String, c UInt64, PROJECTION p1 (SELECT a, sum(c) GROUP BY a, b), PROJECTION p2 (SELECT b, sum(c) GROUP BY b)) ENGINE=MergeTree()"
)
def test_agg_functions(self):
def extract_agg_func(query):
@ -856,3 +874,8 @@ LIFETIME(MIN 0 MAX 0)""",
)
parse_one("foobar(x)").assert_is(exp.Anonymous)
def test_drop_on_cluster(self):
for creatable in ("DATABASE", "TABLE", "VIEW", "DICTIONARY", "FUNCTION"):
with self.subTest(f"Test DROP {creatable} ON CLUSTER"):
self.validate_identity(f"DROP {creatable} test ON CLUSTER test_cluster")

104
tests/dialects/test_dialect.py
View file

@ -1019,6 +1019,19 @@ class TestDialect(Validator):
},
)
self.validate_all(
"TIMESTAMP_TRUNC(x, DAY, 'UTC')",
write={
"": "TIMESTAMP_TRUNC(x, DAY, 'UTC')",
"duckdb": "DATE_TRUNC('DAY', x)",
"presto": "DATE_TRUNC('DAY', x)",
"postgres": "DATE_TRUNC('DAY', x, 'UTC')",
"snowflake": "DATE_TRUNC('DAY', x)",
"databricks": "DATE_TRUNC('DAY', x)",
"clickhouse": "DATE_TRUNC('DAY', x, 'UTC')",
},
)
for unit in ("DAY", "MONTH", "YEAR"):
self.validate_all(
f"{unit}(x)",
@ -1681,6 +1694,26 @@ class TestDialect(Validator):
"tsql": "CAST(a AS FLOAT) / b",
},
)
self.validate_all(
"MOD(8 - 1 + 7, 7)",
write={
"": "(8 - 1 + 7) % 7",
"hive": "(8 - 1 + 7) % 7",
"presto": "(8 - 1 + 7) % 7",
"snowflake": "(8 - 1 + 7) % 7",
"bigquery": "MOD(8 - 1 + 7, 7)",
},
)
self.validate_all(
"MOD(a, b + 1)",
write={
"": "a % (b + 1)",
"hive": "a % (b + 1)",
"presto": "a % (b + 1)",
"snowflake": "a % (b + 1)",
"bigquery": "MOD(a, b + 1)",
},
)
def test_typeddiv(self):
typed_div = exp.Div(this=exp.column("a"), expression=exp.column("b"), typed=True)
@ -2186,6 +2219,8 @@ SELECT
)
def test_cast_to_user_defined_type(self):
self.validate_identity("CAST(x AS some_udt(1234))")
self.validate_all(
"CAST(x AS some_udt)",
write={
@ -2214,6 +2249,18 @@ SELECT
"tsql": "SELECT * FROM (SELECT *, COUNT(*) OVER () AS _w FROM t) AS _t WHERE _w > 1",
},
)
self.validate_all(
'SELECT "user id", some_id, 1 as other_id, 2 as "2 nd id" FROM t QUALIFY COUNT(*) OVER () > 1',
write={
"duckdb": 'SELECT "user id", some_id, 1 AS other_id, 2 AS "2 nd id" FROM t QUALIFY COUNT(*) OVER () > 1',
"snowflake": 'SELECT "user id", some_id, 1 AS other_id, 2 AS "2 nd id" FROM t QUALIFY COUNT(*) OVER () > 1',
"clickhouse": 'SELECT "user id", some_id, other_id, "2 nd id" FROM (SELECT "user id", some_id, 1 AS other_id, 2 AS "2 nd id", COUNT(*) OVER () AS _w FROM t) AS _t WHERE _w > 1',
"mysql": "SELECT `user id`, some_id, other_id, `2 nd id` FROM (SELECT `user id`, some_id, 1 AS other_id, 2 AS `2 nd id`, COUNT(*) OVER () AS _w FROM t) AS _t WHERE _w > 1",
"oracle": 'SELECT "user id", some_id, other_id, "2 nd id" FROM (SELECT "user id", some_id, 1 AS other_id, 2 AS "2 nd id", COUNT(*) OVER () AS _w FROM t) _t WHERE _w > 1',
"postgres": 'SELECT "user id", some_id, other_id, "2 nd id" FROM (SELECT "user id", some_id, 1 AS other_id, 2 AS "2 nd id", COUNT(*) OVER () AS _w FROM t) AS _t WHERE _w > 1',
"tsql": "SELECT [user id], some_id, other_id, [2 nd id] FROM (SELECT [user id] AS [user id], some_id AS some_id, 1 AS other_id, 2 AS [2 nd id], COUNT(*) OVER () AS _w FROM t) AS _t WHERE _w > 1",
},
)
def test_nested_ctes(self):
self.validate_all(
@ -2249,7 +2296,7 @@ SELECT
"WITH t1(x) AS (SELECT 1) SELECT * FROM (WITH t2(y) AS (SELECT 2) SELECT y FROM t2) AS subq",
write={
"duckdb": "WITH t1(x) AS (SELECT 1) SELECT * FROM (WITH t2(y) AS (SELECT 2) SELECT y FROM t2) AS subq",
"tsql": "WITH t2(y) AS (SELECT 2), t1(x) AS (SELECT 1) SELECT * FROM (SELECT y AS y FROM t2) AS subq",
"tsql": "WITH t1(x) AS (SELECT 1), t2(y) AS (SELECT 2) SELECT * FROM (SELECT y AS y FROM t2) AS subq",
},
)
self.validate_all(
@ -2273,6 +2320,59 @@ FROM c""",
"hive": "WITH a1 AS (SELECT 1), a2 AS (SELECT 2), b AS (SELECT * FROM a1, a2), c AS (SELECT * FROM b) SELECT * FROM c",
},
)
self.validate_all(
"""
WITH subquery1 AS (
WITH tmp AS (
SELECT
*
FROM table0
)
SELECT
*
FROM tmp
), subquery2 AS (
WITH tmp2 AS (
SELECT
*
FROM table1
WHERE
a IN subquery1
)
SELECT
*
FROM tmp2
)
SELECT
*
FROM subquery2
""",
write={
"hive": """WITH tmp AS (
SELECT
*
FROM table0
), subquery1 AS (
SELECT
*
FROM tmp
), tmp2 AS (
SELECT
*
FROM table1
WHERE
a IN subquery1
), subquery2 AS (
SELECT
*
FROM tmp2
)
SELECT
*
FROM subquery2""",
},
pretty=True,
)
def test_unsupported_null_ordering(self):
# We'll transpile a portable query from the following dialects to MySQL / T-SQL, which
@ -2372,7 +2472,7 @@ FROM c""",
"hive": UnsupportedError,
"mysql": UnsupportedError,
"oracle": UnsupportedError,
"postgres": "(ARRAY_LENGTH(arr, 1) = 0 OR ARRAY_LENGTH(ARRAY(SELECT x FROM UNNEST(arr) AS _t(x) WHERE pred), 1) <> 0)",
"postgres": "(ARRAY_LENGTH(arr, 1) = 0 OR ARRAY_LENGTH(ARRAY(SELECT x FROM UNNEST(arr) AS _t0(x) WHERE pred), 1) <> 0)",
"presto": "ANY_MATCH(arr, x -> pred)",
"redshift": UnsupportedError,
"snowflake": UnsupportedError,

97
tests/dialects/test_duckdb.py
View file

@ -271,6 +271,14 @@ class TestDuckDB(Validator):
self.validate_identity(
"SELECT * FROM x LEFT JOIN UNNEST(y)", "SELECT * FROM x LEFT JOIN UNNEST(y) ON TRUE"
)
self.validate_identity(
"SELECT JSON_EXTRACT_STRING(c, '$.k1') = 'v1'",
"SELECT (c ->> '$.k1') = 'v1'",
)
self.validate_identity(
"SELECT JSON_EXTRACT(c, '$.k1') = 'v1'",
"SELECT (c -> '$.k1') = 'v1'",
)
self.validate_identity(
"""SELECT '{"foo": [1, 2, 3]}' -> 'foo' -> 0""",
"""SELECT '{"foo": [1, 2, 3]}' -> '$.foo' -> '$[0]'""",
@ -734,6 +742,28 @@ class TestDuckDB(Validator):
)
self.validate_identity("COPY lineitem (l_orderkey) TO 'orderkey.tbl' WITH (DELIMITER '|')")
self.validate_all(
"VARIANCE(a)",
write={
"duckdb": "VARIANCE(a)",
"clickhouse": "varSamp(a)",
},
)
self.validate_all(
"STDDEV(a)",
write={
"duckdb": "STDDEV(a)",
"clickhouse": "stddevSamp(a)",
},
)
self.validate_all(
"DATE_TRUNC('DAY', x)",
write={
"duckdb": "DATE_TRUNC('DAY', x)",
"clickhouse": "DATE_TRUNC('DAY', x)",
},
)
def test_array_index(self):
with self.assertLogs(helper_logger) as cm:
self.validate_all(
@ -803,7 +833,7 @@ class TestDuckDB(Validator):
write={"duckdb": "SELECT (90 * INTERVAL '1' DAY)"},
)
self.validate_all(
"SELECT ((DATE_TRUNC('DAY', CAST(CAST(DATE_TRUNC('DAY', CURRENT_TIMESTAMP) AS DATE) AS TIMESTAMP) + INTERVAL (0 - (DAYOFWEEK(CAST(CAST(DATE_TRUNC('DAY', CURRENT_TIMESTAMP) AS DATE) AS TIMESTAMP)) % 7) - 1 + 7 % 7) DAY) + (7 * INTERVAL (-5) DAY))) AS t1",
"SELECT ((DATE_TRUNC('DAY', CAST(CAST(DATE_TRUNC('DAY', CURRENT_TIMESTAMP) AS DATE) AS TIMESTAMP) + INTERVAL (0 - ((DAYOFWEEK(CAST(CAST(DATE_TRUNC('DAY', CURRENT_TIMESTAMP) AS DATE) AS TIMESTAMP)) % 7) - 1 + 7) % 7) DAY) + (7 * INTERVAL (-5) DAY))) AS t1",
read={
"presto": "SELECT ((DATE_ADD('week', -5, DATE_TRUNC('DAY', DATE_ADD('day', (0 - MOD((DAY_OF_WEEK(CAST(CAST(DATE_TRUNC('DAY', NOW()) AS DATE) AS TIMESTAMP)) % 7) - 1 + 7, 7)), CAST(CAST(DATE_TRUNC('DAY', NOW()) AS DATE) AS TIMESTAMP)))))) AS t1",
},
@ -827,6 +857,9 @@ class TestDuckDB(Validator):
"duckdb": "EPOCH_MS(x)",
"presto": "FROM_UNIXTIME(CAST(x AS DOUBLE) / POW(10, 3))",
"spark": "TIMESTAMP_MILLIS(x)",
"clickhouse": "fromUnixTimestamp64Milli(CAST(x AS Int64))",
"postgres": "TO_TIMESTAMP(CAST(x AS DOUBLE PRECISION) / 10 ^ 3)",
"mysql": "FROM_UNIXTIME(x / POWER(10, 3))",
},
)
self.validate_all(
@ -892,11 +925,11 @@ class TestDuckDB(Validator):
def test_sample(self):
self.validate_identity(
"SELECT * FROM tbl USING SAMPLE 5",
"SELECT * FROM tbl USING SAMPLE (5 ROWS)",
"SELECT * FROM tbl USING SAMPLE RESERVOIR (5 ROWS)",
)
self.validate_identity(
"SELECT * FROM tbl USING SAMPLE 10%",
"SELECT * FROM tbl USING SAMPLE (10 PERCENT)",
"SELECT * FROM tbl USING SAMPLE SYSTEM (10 PERCENT)",
)
self.validate_identity(
"SELECT * FROM tbl USING SAMPLE 10 PERCENT (bernoulli)",
@ -920,14 +953,13 @@ class TestDuckDB(Validator):
)
self.validate_all(
"SELECT * FROM example TABLESAMPLE (3 ROWS) REPEATABLE (82)",
"SELECT * FROM example TABLESAMPLE RESERVOIR (3 ROWS) REPEATABLE (82)",
read={
"duckdb": "SELECT * FROM example TABLESAMPLE (3) REPEATABLE (82)",
"snowflake": "SELECT * FROM example SAMPLE (3 ROWS) SEED (82)",
},
write={
"duckdb": "SELECT * FROM example TABLESAMPLE (3 ROWS) REPEATABLE (82)",
"snowflake": "SELECT * FROM example TABLESAMPLE (3 ROWS) SEED (82)",
"duckdb": "SELECT * FROM example TABLESAMPLE RESERVOIR (3 ROWS) REPEATABLE (82)",
},
)
@ -946,10 +978,6 @@ class TestDuckDB(Validator):
self.validate_identity("CAST(x AS DOUBLE)")
self.validate_identity("CAST(x AS DECIMAL(15, 4))")
self.validate_identity("CAST(x AS STRUCT(number BIGINT))")
self.validate_identity(
"CAST(ROW(1, ROW(1)) AS STRUCT(number BIGINT, row STRUCT(number BIGINT)))"
)
self.validate_identity("CAST(x AS INT64)", "CAST(x AS BIGINT)")
self.validate_identity("CAST(x AS INT32)", "CAST(x AS INT)")
self.validate_identity("CAST(x AS INT16)", "CAST(x AS SMALLINT)")
@ -969,6 +997,32 @@ class TestDuckDB(Validator):
self.validate_identity("CAST(x AS BINARY)", "CAST(x AS BLOB)")
self.validate_identity("CAST(x AS VARBINARY)", "CAST(x AS BLOB)")
self.validate_identity("CAST(x AS LOGICAL)", "CAST(x AS BOOLEAN)")
self.validate_identity(
"CAST(ROW(1, ROW(1)) AS STRUCT(number BIGINT, row STRUCT(number BIGINT)))"
)
self.validate_identity(
"123::CHARACTER VARYING",
"CAST(123 AS TEXT)",
)
self.validate_identity(
"CAST([[STRUCT_PACK(a := 1)]] AS STRUCT(a BIGINT)[][])",
"CAST([[{'a': 1}]] AS STRUCT(a BIGINT)[][])",
)
self.validate_identity(
"CAST([STRUCT_PACK(a := 1)] AS STRUCT(a BIGINT)[])",
"CAST([{'a': 1}] AS STRUCT(a BIGINT)[])",
)
self.validate_all(
"CAST(x AS DECIMAL(38, 0))",
read={
"snowflake": "CAST(x AS NUMBER)",
"duckdb": "CAST(x AS DECIMAL(38, 0))",
},
write={
"snowflake": "CAST(x AS DECIMAL(38, 0))",
},
)
self.validate_all(
"CAST(x AS NUMERIC)",
write={
@ -993,12 +1047,6 @@ class TestDuckDB(Validator):
"tsql": "CAST(x AS BIT)",
},
)
self.validate_all(
"123::CHARACTER VARYING",
write={
"duckdb": "CAST(123 AS TEXT)",
},
)
self.validate_all(
"cast([[1]] as int[][])",
write={
@ -1007,7 +1055,10 @@ class TestDuckDB(Validator):
},
)
self.validate_all(
"CAST(x AS DATE) + INTERVAL (7 * -1) DAY", read={"spark": "DATE_SUB(x, 7)"}
"CAST(x AS DATE) + INTERVAL (7 * -1) DAY",
read={
"spark": "DATE_SUB(x, 7)",
},
)
self.validate_all(
"TRY_CAST(1 AS DOUBLE)",
@ -1034,18 +1085,6 @@ class TestDuckDB(Validator):
"snowflake": "CAST(COL AS ARRAY(BIGINT))",
},
)
self.validate_all(
"CAST([STRUCT_PACK(a := 1)] AS STRUCT(a BIGINT)[])",
write={
"duckdb": "CAST([{'a': 1}] AS STRUCT(a BIGINT)[])",
},
)
self.validate_all(
"CAST([[STRUCT_PACK(a := 1)]] AS STRUCT(a BIGINT)[][])",
write={
"duckdb": "CAST([[{'a': 1}]] AS STRUCT(a BIGINT)[][])",
},
)
def test_bool_or(self):
self.validate_all(

13
tests/dialects/test_mysql.py
View file

@ -148,6 +148,9 @@ class TestMySQL(Validator):
self.validate_identity(
"SELECT * FROM t1, t2, t3 FOR SHARE OF t1 NOWAIT FOR UPDATE OF t2, t3 SKIP LOCKED"
)
self.validate_identity(
"REPLACE INTO table SELECT id FROM table2 WHERE cnt > 100", check_command_warning=True
)
self.validate_identity(
"""SELECT * FROM foo WHERE 3 MEMBER OF(info->'$.value')""",
"""SELECT * FROM foo WHERE 3 MEMBER OF(JSON_EXTRACT(info, '$.value'))""",
@ -607,6 +610,16 @@ class TestMySQL(Validator):
)
def test_mysql(self):
self.validate_all(
"SELECT CONCAT('11', '22')",
read={
"postgres": "SELECT '11' || '22'",
},
write={
"mysql": "SELECT CONCAT('11', '22')",
"postgres": "SELECT CONCAT('11', '22')",
},
)
self.validate_all(
"SELECT department, GROUP_CONCAT(name) AS employee_names FROM data GROUP BY department",
read={

21
tests/dialects/test_postgres.py
View file

@ -733,6 +733,13 @@ class TestPostgres(Validator):
self.validate_identity("TRUNCATE TABLE t1 RESTRICT")
self.validate_identity("TRUNCATE TABLE t1 CONTINUE IDENTITY CASCADE")
self.validate_identity("TRUNCATE TABLE t1 RESTART IDENTITY RESTRICT")
self.validate_identity("ALTER TABLE t1 SET LOGGED")
self.validate_identity("ALTER TABLE t1 SET UNLOGGED")
self.validate_identity("ALTER TABLE t1 SET WITHOUT CLUSTER")
self.validate_identity("ALTER TABLE t1 SET WITHOUT OIDS")
self.validate_identity("ALTER TABLE t1 SET ACCESS METHOD method")
self.validate_identity("ALTER TABLE t1 SET TABLESPACE tablespace")
self.validate_identity("ALTER TABLE t1 SET (fillfactor = 5, autovacuum_enabled = TRUE)")
self.validate_identity(
"CREATE TABLE t (vid INT NOT NULL, CONSTRAINT ht_vid_nid_fid_idx EXCLUDE (INT4RANGE(vid, nid) WITH &&, INT4RANGE(fid, fid, '[]') WITH &&))"
)
@ -797,6 +804,12 @@ class TestPostgres(Validator):
"CREATE TABLE test (x TIMESTAMP WITHOUT TIME ZONE[][])",
"CREATE TABLE test (x TIMESTAMP[][])",
)
self.validate_identity(
"CREATE FUNCTION add(integer, integer) RETURNS INT LANGUAGE SQL IMMUTABLE RETURNS NULL ON NULL INPUT AS 'select $1 + $2;'",
)
self.validate_identity(
"CREATE FUNCTION add(integer, integer) RETURNS INT LANGUAGE SQL IMMUTABLE STRICT AS 'select $1 + $2;'"
)
self.validate_identity(
"CREATE FUNCTION add(INT, INT) RETURNS INT SET search_path TO 'public' AS 'select $1 + $2;' LANGUAGE SQL IMMUTABLE",
check_command_warning=True,
@ -805,18 +818,10 @@ class TestPostgres(Validator):
"CREATE FUNCTION x(INT) RETURNS INT SET foo FROM CURRENT",
check_command_warning=True,
)
self.validate_identity(
"CREATE FUNCTION add(integer, integer) RETURNS integer AS 'select $1 + $2;' LANGUAGE SQL IMMUTABLE RETURNS NULL ON NULL INPUT",
check_command_warning=True,
)
self.validate_identity(
"CREATE FUNCTION add(integer, integer) RETURNS integer AS 'select $1 + $2;' LANGUAGE SQL IMMUTABLE CALLED ON NULL INPUT",
check_command_warning=True,
)
self.validate_identity(
"CREATE FUNCTION add(integer, integer) RETURNS integer AS 'select $1 + $2;' LANGUAGE SQL IMMUTABLE STRICT",
check_command_warning=True,
)
self.validate_identity(
"CREATE CONSTRAINT TRIGGER my_trigger AFTER INSERT OR DELETE OR UPDATE OF col_a, col_b ON public.my_table DEFERRABLE INITIALLY DEFERRED FOR EACH ROW EXECUTE FUNCTION do_sth()",
check_command_warning=True,

20
tests/dialects/test_redshift.py
View file

@ -498,7 +498,25 @@ FROM (
},
)
def test_rename_table(self):
def test_alter_table(self):
self.validate_identity("ALTER TABLE s.t ALTER SORTKEY (c)")
self.validate_identity("ALTER TABLE t ALTER SORTKEY AUTO")
self.validate_identity("ALTER TABLE t ALTER SORTKEY NONE")
self.validate_identity("ALTER TABLE t ALTER SORTKEY (c1, c2)")
self.validate_identity("ALTER TABLE t ALTER SORTKEY (c1, c2)")
self.validate_identity("ALTER TABLE t ALTER COMPOUND SORTKEY (c1, c2)")
self.validate_identity("ALTER TABLE t ALTER DISTSTYLE ALL")
self.validate_identity("ALTER TABLE t ALTER DISTSTYLE EVEN")
self.validate_identity("ALTER TABLE t ALTER DISTSTYLE AUTO")
self.validate_identity("ALTER TABLE t ALTER DISTSTYLE KEY DISTKEY c")
self.validate_identity("ALTER TABLE t SET TABLE PROPERTIES ('a' = '5', 'b' = 'c')")
self.validate_identity("ALTER TABLE t SET LOCATION 's3://bucket/folder/'")
self.validate_identity("ALTER TABLE t SET FILE FORMAT AVRO")
self.validate_identity(
"ALTER TABLE t ALTER DISTKEY c",
"ALTER TABLE t ALTER DISTSTYLE KEY DISTKEY c",
)
self.validate_all(
"ALTER TABLE db.t1 RENAME TO db.t2",
write={

302
tests/dialects/test_snowflake.py
View file

@ -10,14 +10,14 @@ class TestSnowflake(Validator):
dialect = "snowflake"
def test_snowflake(self):
self.validate_identity(
"MERGE INTO my_db AS ids USING (SELECT new_id FROM my_model WHERE NOT col IS NULL) AS new_ids ON ids.type = new_ids.type AND ids.source = new_ids.source WHEN NOT MATCHED THEN INSERT VALUES (new_ids.new_id)"
self.validate_all(
"ARRAY_CONSTRUCT_COMPACT(1, null, 2)",
write={
"spark": "ARRAY_COMPACT(ARRAY(1, NULL, 2))",
"snowflake": "ARRAY_CONSTRUCT_COMPACT(1, NULL, 2)",
},
)
self.validate_identity("ALTER TABLE table1 CLUSTER BY (name DESC)")
self.validate_identity(
"INSERT OVERWRITE TABLE t SELECT 1", "INSERT OVERWRITE INTO t SELECT 1"
)
self.validate_identity("SELECT rename, replace")
expr = parse_one("SELECT APPROX_TOP_K(C4, 3, 5) FROM t")
expr.selects[0].assert_is(exp.AggFunc)
self.assertEqual(expr.sql(dialect="snowflake"), "SELECT APPROX_TOP_K(C4, 3, 5) FROM t")
@ -43,6 +43,8 @@ WHERE
)""",
)
self.validate_identity("ALTER TABLE table1 CLUSTER BY (name DESC)")
self.validate_identity("SELECT rename, replace")
self.validate_identity("SELECT TIMEADD(HOUR, 2, CAST('09:05:03' AS TIME))")
self.validate_identity("SELECT CAST(OBJECT_CONSTRUCT('a', 1) AS MAP(VARCHAR, INT))")
self.validate_identity("SELECT CAST(OBJECT_CONSTRUCT('a', 1) AS OBJECT(a CHAR NOT NULL))")
@ -86,20 +88,19 @@ WHERE
self.validate_identity("a$b") # valid snowflake identifier
self.validate_identity("SELECT REGEXP_LIKE(a, b, c)")
self.validate_identity("CREATE TABLE foo (bar FLOAT AUTOINCREMENT START 0 INCREMENT 1)")
self.validate_identity("ALTER TABLE IF EXISTS foo SET TAG a = 'a', b = 'b', c = 'c'")
self.validate_identity("ALTER TABLE foo UNSET TAG a, b, c")
self.validate_identity("ALTER TABLE foo SET COMMENT = 'bar'")
self.validate_identity("ALTER TABLE foo SET CHANGE_TRACKING = FALSE")
self.validate_identity("ALTER TABLE foo UNSET DATA_RETENTION_TIME_IN_DAYS, CHANGE_TRACKING")
self.validate_identity("COMMENT IF EXISTS ON TABLE foo IS 'bar'")
self.validate_identity("SELECT CONVERT_TIMEZONE('UTC', 'America/Los_Angeles', col)")
self.validate_identity("ALTER TABLE a SWAP WITH b")
self.validate_identity("SELECT MATCH_CONDITION")
self.validate_identity("SELECT * REPLACE (CAST(col AS TEXT) AS scol) FROM t")
self.validate_identity(
'DESCRIBE TABLE "SNOWFLAKE_SAMPLE_DATA"."TPCDS_SF100TCL"."WEB_SITE" type=stage'
"MERGE INTO my_db AS ids USING (SELECT new_id FROM my_model WHERE NOT col IS NULL) AS new_ids ON ids.type = new_ids.type AND ids.source = new_ids.source WHEN NOT MATCHED THEN INSERT VALUES (new_ids.new_id)"
)
self.validate_identity(
"SELECT a FROM test PIVOT(SUM(x) FOR y IN ('z', 'q')) AS x TABLESAMPLE (0.1)"
"INSERT OVERWRITE TABLE t SELECT 1", "INSERT OVERWRITE INTO t SELECT 1"
)
self.validate_identity(
'DESCRIBE TABLE "SNOWFLAKE_SAMPLE_DATA"."TPCDS_SF100TCL"."WEB_SITE" type=stage'
)
self.validate_identity(
"SELECT * FROM DATA AS DATA_L ASOF JOIN DATA AS DATA_R MATCH_CONDITION (DATA_L.VAL > DATA_R.VAL) ON DATA_L.ID = DATA_R.ID"
@ -145,10 +146,6 @@ WHERE
"SELECT TIMESTAMPFROMPARTS(d, t)",
"SELECT TIMESTAMP_FROM_PARTS(d, t)",
)
self.validate_identity(
"SELECT user_id, value FROM table_name SAMPLE ($s) SEED (0)",
"SELECT user_id, value FROM table_name TABLESAMPLE ($s) SEED (0)",
)
self.validate_identity(
"SELECT v:attr[0].name FROM vartab",
"SELECT GET_PATH(v, 'attr[0].name') FROM vartab",
@ -236,6 +233,38 @@ WHERE
"CAST(x AS NCHAR VARYING)",
"CAST(x AS VARCHAR)",
)
self.validate_identity(
"CREATE OR REPLACE TEMPORARY TABLE x (y NUMBER IDENTITY(0, 1))",
"CREATE OR REPLACE TEMPORARY TABLE x (y DECIMAL(38, 0) AUTOINCREMENT START 0 INCREMENT 1)",
)
self.validate_identity(
"CREATE TEMPORARY TABLE x (y NUMBER AUTOINCREMENT(0, 1))",
"CREATE TEMPORARY TABLE x (y DECIMAL(38, 0) AUTOINCREMENT START 0 INCREMENT 1)",
)
self.validate_identity(
"CREATE TABLE x (y NUMBER IDENTITY START 0 INCREMENT 1)",
"CREATE TABLE x (y DECIMAL(38, 0) AUTOINCREMENT START 0 INCREMENT 1)",
)
self.validate_identity(
"ALTER TABLE foo ADD COLUMN id INT identity(1, 1)",
"ALTER TABLE foo ADD COLUMN id INT AUTOINCREMENT START 1 INCREMENT 1",
)
self.validate_identity(
"SELECT DAYOFWEEK('2016-01-02T23:39:20.123-07:00'::TIMESTAMP)",
"SELECT DAYOFWEEK(CAST('2016-01-02T23:39:20.123-07:00' AS TIMESTAMP))",
)
self.validate_identity(
"SELECT * FROM xxx WHERE col ilike '%Don''t%'",
"SELECT * FROM xxx WHERE col ILIKE '%Don\\'t%'",
)
self.validate_identity(
"SELECT * EXCLUDE a, b FROM xxx",
"SELECT * EXCLUDE (a), b FROM xxx",
)
self.validate_identity(
"SELECT * RENAME a AS b, c AS d FROM xxx",
"SELECT * RENAME (a AS b), c AS d FROM xxx",
)
self.validate_all(
"OBJECT_CONSTRUCT_KEEP_NULL('key_1', 'one', 'key_2', NULL)",
@ -249,18 +278,6 @@ WHERE
"snowflake": "OBJECT_CONSTRUCT_KEEP_NULL('key_1', 'one', 'key_2', NULL)",
},
)
self.validate_all(
"SELECT * FROM example TABLESAMPLE (3) SEED (82)",
read={
"databricks": "SELECT * FROM example TABLESAMPLE (3 PERCENT) REPEATABLE (82)",
"duckdb": "SELECT * FROM example TABLESAMPLE (3 PERCENT) REPEATABLE (82)",
},
write={
"databricks": "SELECT * FROM example TABLESAMPLE (3 PERCENT) REPEATABLE (82)",
"duckdb": "SELECT * FROM example TABLESAMPLE (3 PERCENT) REPEATABLE (82)",
"snowflake": "SELECT * FROM example TABLESAMPLE (3) SEED (82)",
},
)
self.validate_all(
"SELECT TIME_FROM_PARTS(12, 34, 56, 987654321)",
write={
@ -568,60 +585,12 @@ WHERE
},
)
self.validate_all(
"CREATE OR REPLACE TEMPORARY TABLE x (y NUMBER IDENTITY(0, 1))",
write={
"snowflake": "CREATE OR REPLACE TEMPORARY TABLE x (y DECIMAL AUTOINCREMENT START 0 INCREMENT 1)",
},
)
self.validate_all(
"CREATE TEMPORARY TABLE x (y NUMBER AUTOINCREMENT(0, 1))",
write={
"snowflake": "CREATE TEMPORARY TABLE x (y DECIMAL AUTOINCREMENT START 0 INCREMENT 1)",
},
)
self.validate_all(
"CREATE TABLE x (y NUMBER IDENTITY START 0 INCREMENT 1)",
write={
"snowflake": "CREATE TABLE x (y DECIMAL AUTOINCREMENT START 0 INCREMENT 1)",
},
)
self.validate_all(
"ALTER TABLE foo ADD COLUMN id INT identity(1, 1)",
write={
"snowflake": "ALTER TABLE foo ADD COLUMN id INT AUTOINCREMENT START 1 INCREMENT 1",
},
)
self.validate_all(
"SELECT DAYOFWEEK('2016-01-02T23:39:20.123-07:00'::TIMESTAMP)",
write={
"snowflake": "SELECT DAYOFWEEK(CAST('2016-01-02T23:39:20.123-07:00' AS TIMESTAMP))",
},
)
self.validate_all(
"SELECT * FROM xxx WHERE col ilike '%Don''t%'",
write={
"snowflake": "SELECT * FROM xxx WHERE col ILIKE '%Don\\'t%'",
},
)
self.validate_all(
"SELECT * EXCLUDE a, b FROM xxx",
write={
"snowflake": "SELECT * EXCLUDE (a), b FROM xxx",
},
)
self.validate_all(
"SELECT * RENAME a AS b, c AS d FROM xxx",
write={
"snowflake": "SELECT * RENAME (a AS b), c AS d FROM xxx",
},
)
self.validate_all(
"SELECT * EXCLUDE (a, b) RENAME (c AS d, E AS F) FROM xxx",
"SELECT * EXCLUDE (a, b) REPLACE (c AS d, E AS F) FROM xxx",
read={
"duckdb": "SELECT * EXCLUDE (a, b) REPLACE (c AS d, E AS F) FROM xxx",
},
write={
"snowflake": "SELECT * EXCLUDE (a, b) RENAME (c AS d, E AS F) FROM xxx",
"snowflake": "SELECT * EXCLUDE (a, b) REPLACE (c AS d, E AS F) FROM xxx",
"duckdb": "SELECT * EXCLUDE (a, b) REPLACE (c AS d, E AS F) FROM xxx",
},
)
@ -831,6 +800,18 @@ WHERE
"snowflake": "SELECT LISTAGG(col1, ', ') WITHIN GROUP (ORDER BY col2) FROM t",
},
)
self.validate_all(
"SELECT APPROX_PERCENTILE(a, 0.5) FROM t",
read={
"trino": "SELECT APPROX_PERCENTILE(a, 1, 0.5, 0.001) FROM t",
"presto": "SELECT APPROX_PERCENTILE(a, 1, 0.5, 0.001) FROM t",
},
write={
"trino": "SELECT APPROX_PERCENTILE(a, 0.5) FROM t",
"presto": "SELECT APPROX_PERCENTILE(a, 0.5) FROM t",
"snowflake": "SELECT APPROX_PERCENTILE(a, 0.5) FROM t",
},
)
def test_null_treatment(self):
self.validate_all(
@ -881,6 +862,10 @@ WHERE
self.validate_identity("SELECT * FROM @namespace.%table_name/path/to/file.json.gz")
self.validate_identity("SELECT * FROM '@external/location' (FILE_FORMAT => 'path.to.csv')")
self.validate_identity("PUT file:///dir/tmp.csv @%table", check_command_warning=True)
self.validate_identity("SELECT * FROM (SELECT a FROM @foo)")
self.validate_identity(
"SELECT * FROM (SELECT * FROM '@external/location' (FILE_FORMAT => 'path.to.csv'))"
)
self.validate_identity(
"SELECT * FROM @foo/bar (FILE_FORMAT => ds_sandbox.test.my_csv_format, PATTERN => 'test') AS bla"
)
@ -902,18 +887,27 @@ WHERE
def test_sample(self):
self.validate_identity("SELECT * FROM testtable TABLESAMPLE BERNOULLI (20.3)")
self.validate_identity("SELECT * FROM testtable TABLESAMPLE (100)")
self.validate_identity("SELECT * FROM testtable TABLESAMPLE SYSTEM (3) SEED (82)")
self.validate_identity("SELECT * FROM testtable TABLESAMPLE (10 ROWS)")
self.validate_identity(
"SELECT i, j FROM table1 AS t1 INNER JOIN table2 AS t2 TABLESAMPLE (50) WHERE t2.j = t1.i"
"SELECT a FROM test PIVOT(SUM(x) FOR y IN ('z', 'q')) AS x TABLESAMPLE BERNOULLI (0.1)"
)
self.validate_identity(
"SELECT * FROM (SELECT * FROM t1 JOIN t2 ON t1.a = t2.c) TABLESAMPLE (1)"
"SELECT i, j FROM table1 AS t1 INNER JOIN table2 AS t2 TABLESAMPLE BERNOULLI (50) WHERE t2.j = t1.i"
)
self.validate_identity(
"SELECT * FROM (SELECT * FROM t1 JOIN t2 ON t1.a = t2.c) TABLESAMPLE BERNOULLI (1)"
)
self.validate_identity(
"SELECT * FROM testtable TABLESAMPLE (10 ROWS)",
"SELECT * FROM testtable TABLESAMPLE BERNOULLI (10 ROWS)",
)
self.validate_identity(
"SELECT * FROM testtable TABLESAMPLE (100)",
"SELECT * FROM testtable TABLESAMPLE BERNOULLI (100)",
)
self.validate_identity(
"SELECT * FROM testtable SAMPLE (10)",
"SELECT * FROM testtable TABLESAMPLE (10)",
"SELECT * FROM testtable TABLESAMPLE BERNOULLI (10)",
)
self.validate_identity(
"SELECT * FROM testtable SAMPLE ROW (0)",
@ -923,7 +917,29 @@ WHERE
"SELECT a FROM test SAMPLE BLOCK (0.5) SEED (42)",
"SELECT a FROM test TABLESAMPLE BLOCK (0.5) SEED (42)",
)
self.validate_identity(
"SELECT user_id, value FROM table_name SAMPLE BERNOULLI ($s) SEED (0)",
"SELECT user_id, value FROM table_name TABLESAMPLE BERNOULLI ($s) SEED (0)",
)
self.validate_all(
"SELECT * FROM example TABLESAMPLE BERNOULLI (3) SEED (82)",
read={
"duckdb": "SELECT * FROM example TABLESAMPLE BERNOULLI (3 PERCENT) REPEATABLE (82)",
},
write={
"databricks": "SELECT * FROM example TABLESAMPLE (3 PERCENT) REPEATABLE (82)",
"duckdb": "SELECT * FROM example TABLESAMPLE BERNOULLI (3 PERCENT) REPEATABLE (82)",
"snowflake": "SELECT * FROM example TABLESAMPLE BERNOULLI (3) SEED (82)",
},
)
self.validate_all(
"SELECT * FROM test AS _tmp TABLESAMPLE (5)",
write={
"postgres": "SELECT * FROM test AS _tmp TABLESAMPLE BERNOULLI (5)",
"snowflake": "SELECT * FROM test AS _tmp TABLESAMPLE BERNOULLI (5)",
},
)
self.validate_all(
"""
SELECT i, j
@ -933,7 +949,7 @@ WHERE
table2 AS t2 SAMPLE (50) -- 50% of rows in table2
WHERE t2.j = t1.i""",
write={
"snowflake": "SELECT i, j FROM table1 AS t1 TABLESAMPLE (25) /* 25% of rows in table1 */ INNER JOIN table2 AS t2 TABLESAMPLE (50) /* 50% of rows in table2 */ WHERE t2.j = t1.i",
"snowflake": "SELECT i, j FROM table1 AS t1 TABLESAMPLE BERNOULLI (25) /* 25% of rows in table1 */ INNER JOIN table2 AS t2 TABLESAMPLE BERNOULLI (50) /* 50% of rows in table2 */ WHERE t2.j = t1.i",
},
)
self.validate_all(
@ -945,7 +961,7 @@ WHERE
self.validate_all(
"SELECT * FROM (SELECT * FROM t1 join t2 on t1.a = t2.c) SAMPLE (1)",
write={
"snowflake": "SELECT * FROM (SELECT * FROM t1 JOIN t2 ON t1.a = t2.c) TABLESAMPLE (1)",
"snowflake": "SELECT * FROM (SELECT * FROM t1 JOIN t2 ON t1.a = t2.c) TABLESAMPLE BERNOULLI (1)",
"spark": "SELECT * FROM (SELECT * FROM t1 JOIN t2 ON t1.a = t2.c) TABLESAMPLE (1 PERCENT)",
},
)
@ -1172,7 +1188,7 @@ WHERE
location=@s2/logs/
partition_type = user_specified
file_format = (type = parquet)""",
"CREATE EXTERNAL TABLE et2 (col1 DATE AS (CAST(GET_PATH(PARSE_JSON(metadata$external_table_partition), 'COL1') AS DATE)), col2 VARCHAR AS (CAST(GET_PATH(PARSE_JSON(metadata$external_table_partition), 'COL2') AS VARCHAR)), col3 DECIMAL AS (CAST(GET_PATH(PARSE_JSON(metadata$external_table_partition), 'COL3') AS DECIMAL))) LOCATION @s2/logs/ PARTITION BY (col1, col2, col3) partition_type=user_specified file_format=(type = parquet)",
"CREATE EXTERNAL TABLE et2 (col1 DATE AS (CAST(GET_PATH(PARSE_JSON(metadata$external_table_partition), 'COL1') AS DATE)), col2 VARCHAR AS (CAST(GET_PATH(PARSE_JSON(metadata$external_table_partition), 'COL2') AS VARCHAR)), col3 DECIMAL(38, 0) AS (CAST(GET_PATH(PARSE_JSON(metadata$external_table_partition), 'COL3') AS DECIMAL(38, 0)))) LOCATION @s2/logs/ PARTITION BY (col1, col2, col3) partition_type=user_specified file_format=(type = parquet)",
)
self.validate_identity("CREATE OR REPLACE VIEW foo (uid) COPY GRANTS AS (SELECT 1)")
self.validate_identity("CREATE TABLE geospatial_table (id INT, g GEOGRAPHY)")
@ -1181,8 +1197,12 @@ WHERE
self.validate_identity("CREATE SCHEMA mytestschema_clone CLONE testschema")
self.validate_identity("CREATE TABLE IDENTIFIER('foo') (COLUMN1 VARCHAR, COLUMN2 VARCHAR)")
self.validate_identity("CREATE TABLE IDENTIFIER($foo) (col1 VARCHAR, col2 VARCHAR)")
self.validate_identity("CREATE TAG cost_center ALLOWED_VALUES 'a', 'b'")
self.validate_identity(
"DROP function my_udf (OBJECT(city VARCHAR, zipcode DECIMAL, val ARRAY(BOOLEAN)))"
"CREATE OR REPLACE TAG IF NOT EXISTS cost_center COMMENT='cost_center tag'"
).this.assert_is(exp.Identifier)
self.validate_identity(
"DROP FUNCTION my_udf (OBJECT(city VARCHAR, zipcode DECIMAL(38, 0), val ARRAY(BOOLEAN)))"
)
self.validate_identity(
"CREATE TABLE orders_clone_restore CLONE orders AT (TIMESTAMP => TO_TIMESTAMP_TZ('04/05/2013 01:02:03', 'mm/dd/yyyy hh24:mi:ss'))"
@ -1203,13 +1223,27 @@ WHERE
"CREATE ICEBERG TABLE my_iceberg_table (amount ARRAY(INT)) CATALOG='SNOWFLAKE' EXTERNAL_VOLUME='my_external_volume' BASE_LOCATION='my/relative/path/from/extvol'"
)
self.validate_identity(
"CREATE OR REPLACE FUNCTION my_udf(location OBJECT(city VARCHAR, zipcode DECIMAL, val ARRAY(BOOLEAN))) RETURNS VARCHAR AS $$ SELECT 'foo' $$",
"CREATE OR REPLACE FUNCTION my_udf(location OBJECT(city VARCHAR, zipcode DECIMAL, val ARRAY(BOOLEAN))) RETURNS VARCHAR AS ' SELECT \\'foo\\' '",
"""CREATE OR REPLACE FUNCTION ibis_udfs.public.object_values("obj" OBJECT) RETURNS ARRAY LANGUAGE JAVASCRIPT RETURNS NULL ON NULL INPUT AS ' return Object.values(obj) '"""
)
self.validate_identity(
"""CREATE OR REPLACE FUNCTION ibis_udfs.public.object_values("obj" OBJECT) RETURNS ARRAY LANGUAGE JAVASCRIPT STRICT AS ' return Object.values(obj) '"""
)
self.validate_identity(
"CREATE OR REPLACE FUNCTION my_udf(location OBJECT(city VARCHAR, zipcode DECIMAL(38, 0), val ARRAY(BOOLEAN))) RETURNS VARCHAR AS $$ SELECT 'foo' $$",
"CREATE OR REPLACE FUNCTION my_udf(location OBJECT(city VARCHAR, zipcode DECIMAL(38, 0), val ARRAY(BOOLEAN))) RETURNS VARCHAR AS ' SELECT \\'foo\\' '",
)
self.validate_identity(
"CREATE OR REPLACE FUNCTION my_udtf(foo BOOLEAN) RETURNS TABLE(col1 ARRAY(INT)) AS $$ WITH t AS (SELECT CAST([1, 2, 3] AS ARRAY(INT)) AS c) SELECT c FROM t $$",
"CREATE OR REPLACE FUNCTION my_udtf(foo BOOLEAN) RETURNS TABLE (col1 ARRAY(INT)) AS ' WITH t AS (SELECT CAST([1, 2, 3] AS ARRAY(INT)) AS c) SELECT c FROM t '",
)
self.validate_identity(
"CREATE SEQUENCE seq1 WITH START=1, INCREMENT=1 ORDER",
"CREATE SEQUENCE seq1 START=1 INCREMENT BY 1 ORDER",
)
self.validate_identity(
"CREATE SEQUENCE seq1 WITH START=1 INCREMENT=1 ORDER",
"CREATE SEQUENCE seq1 START=1 INCREMENT=1 ORDER",
)
self.validate_all(
"CREATE TABLE orders_clone CLONE orders",
@ -1435,6 +1469,9 @@ FROM persons AS p, LATERAL FLATTEN(input => p.c, path => 'contact') AS _flattene
)
def test_values(self):
select = exp.select("*").from_("values (map(['a'], [1]))")
self.assertEqual(select.sql("snowflake"), "SELECT * FROM (SELECT OBJECT_CONSTRUCT('a', 1))")
self.validate_all(
'SELECT "c0", "c1" FROM (VALUES (1, 2), (3, 4)) AS "t0"("c0", "c1")',
read={
@ -1832,10 +1869,7 @@ STORAGE_ALLOWED_LOCATIONS=('s3://mybucket1/path1/', 's3://mybucket2/path2/')""",
self.assertEqual(expression.sql(dialect="snowflake"), "SELECT TRY_CAST(FOO() AS TEXT)")
def test_copy(self):
self.validate_identity(
"""COPY INTO mytable (col1, col2) FROM 's3://mybucket/data/files' FILES = ('file1', 'file2') PATTERN = 'pattern' file_format = (FORMAT_NAME = my_csv_format NULL_IF = ('str1', 'str2')) PARSE_HEADER = TRUE""",
"""COPY INTO mytable (col1, col2) FROM 's3://mybucket/data/files' FILES = ('file1', 'file2') PATTERN = 'pattern' FILE_FORMAT = (FORMAT_NAME = my_csv_format NULL_IF = ('str1', 'str2')) PARSE_HEADER = TRUE""",
)
self.validate_identity("COPY INTO test (c1) FROM (SELECT $1.c1 FROM @mystage)")
self.validate_identity(
"""COPY INTO temp FROM @random_stage/path/ FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER = '|' NULL_IF = () FIELD_OPTIONALLY_ENCLOSED_BY = '"' TIMESTAMP_FORMAT = 'TZHTZM YYYY-MM-DD HH24:MI:SS.FF9' DATE_FORMAT = 'TZHTZM YYYY-MM-DD HH24:MI:SS.FF9' BINARY_FORMAT = BASE64) VALIDATION_MODE = 'RETURN_3_ROWS'"""
)
@ -1845,3 +1879,81 @@ STORAGE_ALLOWED_LOCATIONS=('s3://mybucket1/path1/', 's3://mybucket2/path2/')""",
self.validate_identity(
"""COPY INTO mytable FROM 'azure://myaccount.blob.core.windows.net/mycontainer/data/files' CREDENTIALS = (AZURE_SAS_TOKEN = 'token') ENCRYPTION = (TYPE = 'AZURE_CSE' MASTER_KEY = 'kPx...') FILE_FORMAT = (FORMAT_NAME = my_csv_format)"""
)
self.validate_identity(
"""COPY INTO mytable (col1, col2) FROM 's3://mybucket/data/files' FILES = ('file1', 'file2') PATTERN = 'pattern' FILE_FORMAT = (FORMAT_NAME = my_csv_format NULL_IF = ('str1', 'str2')) PARSE_HEADER = TRUE"""
)
self.validate_all(
"""COPY INTO 's3://example/data.csv'
FROM EXTRA.EXAMPLE.TABLE
credentials = (x)
STORAGE_INTEGRATION = S3_INTEGRATION
FILE_FORMAT = (TYPE = CSV COMPRESSION = NONE NULL_IF = ('') FIELD_OPTIONALLY_ENCLOSED_BY = '"')
HEADER = TRUE
OVERWRITE = TRUE
SINGLE = TRUE
""",
write={
"": """COPY INTO 's3://example/data.csv'
FROM EXTRA.EXAMPLE.TABLE
CREDENTIALS = (x) WITH (
STORAGE_INTEGRATION S3_INTEGRATION,
FILE_FORMAT = (TYPE = CSV COMPRESSION = NONE NULL_IF = (
''
) FIELD_OPTIONALLY_ENCLOSED_BY = '"'),
HEADER TRUE,
OVERWRITE TRUE,
SINGLE TRUE
)""",
"snowflake": """COPY INTO 's3://example/data.csv'
FROM EXTRA.EXAMPLE.TABLE
CREDENTIALS = (x)
STORAGE_INTEGRATION = S3_INTEGRATION
FILE_FORMAT = (TYPE = CSV COMPRESSION = NONE NULL_IF = (
''
) FIELD_OPTIONALLY_ENCLOSED_BY = '"')
HEADER = TRUE
OVERWRITE = TRUE
SINGLE = TRUE""",
},
pretty=True,
)
self.validate_all(
"""COPY INTO 's3://example/data.csv'
FROM EXTRA.EXAMPLE.TABLE
credentials = (x)
STORAGE_INTEGRATION = S3_INTEGRATION
FILE_FORMAT = (TYPE = CSV COMPRESSION = NONE NULL_IF = ('') FIELD_OPTIONALLY_ENCLOSED_BY = '"')
HEADER = TRUE
OVERWRITE = TRUE
SINGLE = TRUE
""",
write={
"": """COPY INTO 's3://example/data.csv' FROM EXTRA.EXAMPLE.TABLE CREDENTIALS = (x) WITH (STORAGE_INTEGRATION S3_INTEGRATION, FILE_FORMAT = (TYPE = CSV COMPRESSION = NONE NULL_IF = ('') FIELD_OPTIONALLY_ENCLOSED_BY = '"'), HEADER TRUE, OVERWRITE TRUE, SINGLE TRUE)""",
"snowflake": """COPY INTO 's3://example/data.csv' FROM EXTRA.EXAMPLE.TABLE CREDENTIALS = (x) STORAGE_INTEGRATION = S3_INTEGRATION FILE_FORMAT = (TYPE = CSV COMPRESSION = NONE NULL_IF = ('') FIELD_OPTIONALLY_ENCLOSED_BY = '"') HEADER = TRUE OVERWRITE = TRUE SINGLE = TRUE""",
},
)
def test_querying_semi_structured_data(self):
self.validate_identity("SELECT $1")
self.validate_identity("SELECT $1.elem")
self.validate_identity("SELECT $1:a.b", "SELECT GET_PATH($1, 'a.b')")
self.validate_identity("SELECT t.$23:a.b", "SELECT GET_PATH(t.$23, 'a.b')")
self.validate_identity("SELECT t.$17:a[0].b[0].c", "SELECT GET_PATH(t.$17, 'a[0].b[0].c')")
def test_alter_set_unset(self):
self.validate_identity("ALTER TABLE tbl SET DATA_RETENTION_TIME_IN_DAYS=1")
self.validate_identity("ALTER TABLE tbl SET DEFAULT_DDL_COLLATION='test'")
self.validate_identity("ALTER TABLE foo SET COMMENT='bar'")
self.validate_identity("ALTER TABLE foo SET CHANGE_TRACKING=FALSE")
self.validate_identity("ALTER TABLE table1 SET TAG foo.bar = 'baz'")
self.validate_identity("ALTER TABLE IF EXISTS foo SET TAG a = 'a', b = 'b', c = 'c'")
self.validate_identity(
"""ALTER TABLE tbl SET STAGE_FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER = '|' NULL_IF = () FIELD_OPTIONALLY_ENCLOSED_BY = '"' TIMESTAMP_FORMAT = 'TZHTZM YYYY-MM-DD HH24:MI:SS.FF9' DATE_FORMAT = 'TZHTZM YYYY-MM-DD HH24:MI:SS.FF9' BINARY_FORMAT = BASE64)""",
)
self.validate_identity(
"""ALTER TABLE tbl SET STAGE_COPY_OPTIONS = (ON_ERROR = SKIP_FILE SIZE_LIMIT = 5 PURGE = TRUE MATCH_BY_COLUMN_NAME = CASE_SENSITIVE)"""
)
self.validate_identity("ALTER TABLE foo UNSET TAG a, b, c")
self.validate_identity("ALTER TABLE foo UNSET DATA_RETENTION_TIME_IN_DAYS, CHANGE_TRACKING")

112
tests/dialects/test_tsql.py
View file

@ -1,5 +1,4 @@
from sqlglot import exp, parse, parse_one
from sqlglot.parser import logger as parser_logger
from tests.dialects.test_dialect import Validator
from sqlglot.errors import ParseError
@ -8,6 +7,8 @@ class TestTSQL(Validator):
dialect = "tsql"
def test_tsql(self):
self.validate_identity("CREATE view a.b.c", "CREATE VIEW b.c")
self.validate_identity("DROP view a.b.c", "DROP VIEW b.c")
self.validate_identity("ROUND(x, 1, 0)")
self.validate_identity("EXEC MyProc @id=7, @name='Lochristi'", check_command_warning=True)
# https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/ms187879(v=sql.105)?redirectedfrom=MSDN
@ -191,16 +192,9 @@ class TestTSQL(Validator):
)
self.validate_all(
"""
CREATE TABLE x(
[zip_cd] [varchar](5) NULL NOT FOR REPLICATION,
[zip_cd_mkey] [varchar](5) NOT NULL,
CONSTRAINT [pk_mytable] PRIMARY KEY CLUSTERED ([zip_cd_mkey] ASC)
WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF) ON [INDEX]
) ON [SECONDARY]
""",
"""CREATE TABLE x ([zip_cd] VARCHAR(5) NULL NOT FOR REPLICATION, [zip_cd_mkey] VARCHAR(5) NOT NULL, CONSTRAINT [pk_mytable] PRIMARY KEY CLUSTERED ([zip_cd_mkey] ASC) WITH (PAD_INDEX=ON, STATISTICS_NORECOMPUTE=OFF) ON [INDEX]) ON [SECONDARY]""",
write={
"tsql": "CREATE TABLE x ([zip_cd] VARCHAR(5) NULL NOT FOR REPLICATION, [zip_cd_mkey] VARCHAR(5) NOT NULL, CONSTRAINT [pk_mytable] PRIMARY KEY CLUSTERED ([zip_cd_mkey] ASC) WITH (PAD_INDEX=ON, STATISTICS_NORECOMPUTE=OFF) ON [INDEX]) ON [SECONDARY]",
"tsql": "CREATE TABLE x ([zip_cd] VARCHAR(5) NULL NOT FOR REPLICATION, [zip_cd_mkey] VARCHAR(5) NOT NULL, CONSTRAINT [pk_mytable] PRIMARY KEY CLUSTERED ([zip_cd_mkey] ASC) WITH (PAD_INDEX=ON, STATISTICS_NORECOMPUTE=OFF) ON [INDEX]) ON [SECONDARY]",
"spark2": "CREATE TABLE x (`zip_cd` VARCHAR(5), `zip_cd_mkey` VARCHAR(5) NOT NULL, CONSTRAINT `pk_mytable` PRIMARY KEY (`zip_cd_mkey`))",
},
)
@ -259,7 +253,7 @@ class TestTSQL(Validator):
self.validate_identity("SELECT * FROM ##foo")
self.validate_identity("SELECT a = 1", "SELECT 1 AS a")
self.validate_identity(
"DECLARE @TestVariable AS VARCHAR(100)='Save Our Planet'", check_command_warning=True
"DECLARE @TestVariable AS VARCHAR(100) = 'Save Our Planet'",
)
self.validate_identity(
"SELECT a = 1 UNION ALL SELECT a = b", "SELECT 1 AS a UNION ALL SELECT b AS a"
@ -461,6 +455,7 @@ class TestTSQL(Validator):
self.validate_identity("CAST(x AS IMAGE)")
self.validate_identity("CAST(x AS SQL_VARIANT)")
self.validate_identity("CAST(x AS BIT)")
self.validate_all(
"CAST(x AS DATETIME2)",
read={
@ -488,7 +483,7 @@ class TestTSQL(Validator):
},
)
def test__types_ints(self):
def test_types_ints(self):
self.validate_all(
"CAST(X AS INT)",
write={
@ -521,10 +516,14 @@ class TestTSQL(Validator):
self.validate_all(
"CAST(X AS TINYINT)",
read={
"duckdb": "CAST(X AS UTINYINT)",
},
write={
"hive": "CAST(X AS TINYINT)",
"spark2": "CAST(X AS TINYINT)",
"spark": "CAST(X AS TINYINT)",
"duckdb": "CAST(X AS UTINYINT)",
"hive": "CAST(X AS SMALLINT)",
"spark2": "CAST(X AS SMALLINT)",
"spark": "CAST(X AS SMALLINT)",
"tsql": "CAST(X AS TINYINT)",
},
)
@ -764,18 +763,32 @@ class TestTSQL(Validator):
expression.sql(dialect="tsql"), "ALTER TABLE dbo.DocExe DROP CONSTRAINT FK_Column_B"
)
for clusterd_keyword in ("CLUSTERED", "NONCLUSTERED"):
for clustered_keyword in ("CLUSTERED", "NONCLUSTERED"):
self.validate_identity(
'CREATE TABLE "dbo"."benchmark" ('
'"name" CHAR(7) NOT NULL, '
'"internal_id" VARCHAR(10) NOT NULL, '
f'UNIQUE {clusterd_keyword} ("internal_id" ASC))',
f'UNIQUE {clustered_keyword} ("internal_id" ASC))',
"CREATE TABLE [dbo].[benchmark] ("
"[name] CHAR(7) NOT NULL, "
"[internal_id] VARCHAR(10) NOT NULL, "
f"UNIQUE {clusterd_keyword} ([internal_id] ASC))",
f"UNIQUE {clustered_keyword} ([internal_id] ASC))",
)
self.validate_identity(
"ALTER TABLE tbl SET SYSTEM_VERSIONING=ON(HISTORY_TABLE=db.tbl, DATA_CONSISTENCY_CHECK=OFF, HISTORY_RETENTION_PERIOD=5 DAYS)"
)
self.validate_identity(
"ALTER TABLE tbl SET SYSTEM_VERSIONING=ON(HISTORY_TABLE=db.tbl, HISTORY_RETENTION_PERIOD=INFINITE)"
)
self.validate_identity("ALTER TABLE tbl SET SYSTEM_VERSIONING=OFF")
self.validate_identity("ALTER TABLE tbl SET FILESTREAM_ON = 'test'")
self.validate_identity(
"ALTER TABLE tbl SET DATA_DELETION=ON(FILTER_COLUMN=col, RETENTION_PERIOD=5 MONTHS)"
)
self.validate_identity("ALTER TABLE tbl SET DATA_DELETION=ON")
self.validate_identity("ALTER TABLE tbl SET DATA_DELETION=OFF")
self.validate_identity(
"CREATE PROCEDURE foo AS BEGIN DELETE FROM bla WHERE foo < CURRENT_TIMESTAMP - 7 END",
"CREATE PROCEDURE foo AS BEGIN DELETE FROM bla WHERE foo < GETDATE() - 7 END",
@ -900,8 +913,7 @@ class TestTSQL(Validator):
def test_udf(self):
self.validate_identity(
"DECLARE @DWH_DateCreated DATETIME = CONVERT(DATETIME, getdate(), 104)",
check_command_warning=True,
"DECLARE @DWH_DateCreated AS DATETIME2 = CONVERT(DATETIME2, GETDATE(), 104)",
)
self.validate_identity(
"CREATE PROCEDURE foo @a INTEGER, @b INTEGER AS SELECT @a = SUM(bla) FROM baz AS bar"
@ -973,9 +985,9 @@ WHERE
BEGIN
SET XACT_ABORT ON;
DECLARE @DWH_DateCreated DATETIME = CONVERT(DATETIME, getdate(), 104);
DECLARE @DWH_DateModified DATETIME = CONVERT(DATETIME, getdate(), 104);
DECLARE @DWH_IdUserCreated INTEGER = SUSER_ID (SYSTEM_USER);
DECLARE @DWH_DateCreated AS DATETIME = CONVERT(DATETIME, getdate(), 104);
DECLARE @DWH_DateModified DATETIME2 = CONVERT(DATETIME2, GETDATE(), 104);
DECLARE @DWH_IdUserCreated INTEGER = SUSER_ID (CURRENT_USER());
DECLARE @DWH_IdUserModified INTEGER = SUSER_ID (SYSTEM_USER);
DECLARE @SalesAmountBefore float;
@ -985,18 +997,17 @@ WHERE
expected_sqls = [
"CREATE PROCEDURE [TRANSF].[SP_Merge_Sales_Real] @Loadid INTEGER, @NumberOfRows INTEGER AS BEGIN SET XACT_ABORT ON",
"DECLARE @DWH_DateCreated DATETIME = CONVERT(DATETIME, getdate(), 104)",
"DECLARE @DWH_DateModified DATETIME = CONVERT(DATETIME, getdate(), 104)",
"DECLARE @DWH_IdUserCreated INTEGER = SUSER_ID (SYSTEM_USER)",
"DECLARE @DWH_IdUserModified INTEGER = SUSER_ID (SYSTEM_USER)",
"DECLARE @SalesAmountBefore float",
"DECLARE @DWH_DateCreated AS DATETIME2 = CONVERT(DATETIME2, GETDATE(), 104)",
"DECLARE @DWH_DateModified AS DATETIME2 = CONVERT(DATETIME2, GETDATE(), 104)",
"DECLARE @DWH_IdUserCreated AS INTEGER = SUSER_ID(CURRENT_USER())",
"DECLARE @DWH_IdUserModified AS INTEGER = SUSER_ID(CURRENT_USER())",
"DECLARE @SalesAmountBefore AS FLOAT",
"SELECT @SalesAmountBefore = SUM(SalesAmount) FROM TRANSF.[Pre_Merge_Sales_Real] AS S",
"END",
]
with self.assertLogs(parser_logger):
for expr, expected_sql in zip(parse(sql, read="tsql"), expected_sqls):
self.assertEqual(expr.sql(dialect="tsql"), expected_sql)
for expr, expected_sql in zip(parse(sql, read="tsql"), expected_sqls):
self.assertEqual(expr.sql(dialect="tsql"), expected_sql)
sql = """
CREATE PROC [dbo].[transform_proc] AS
@ -1010,14 +1021,13 @@ WHERE
"""
expected_sqls = [
"CREATE PROC [dbo].[transform_proc] AS DECLARE @CurrentDate VARCHAR(20)",
"CREATE PROC [dbo].[transform_proc] AS DECLARE @CurrentDate AS VARCHAR(20)",
"SET @CurrentDate = CONVERT(VARCHAR(20), GETDATE(), 120)",
"CREATE TABLE [target_schema].[target_table] (a INTEGER) WITH (DISTRIBUTION=REPLICATE, HEAP)",
]
with self.assertLogs(parser_logger):
for expr, expected_sql in zip(parse(sql, read="tsql"), expected_sqls):
self.assertEqual(expr.sql(dialect="tsql"), expected_sql)
for expr, expected_sql in zip(parse(sql, read="tsql"), expected_sqls):
self.assertEqual(expr.sql(dialect="tsql"), expected_sql)
def test_charindex(self):
self.validate_identity(
@ -1823,3 +1833,35 @@ FROM OPENJSON(@json) WITH (
"duckdb": "WITH t1(c) AS (SELECT 1), t2 AS (SELECT CAST(c AS INTEGER) FROM t1) SELECT * FROM t2",
},
)
def test_declare(self):
# supported cases
self.validate_identity("DECLARE @X INT", "DECLARE @X AS INTEGER")
self.validate_identity("DECLARE @X INT = 1", "DECLARE @X AS INTEGER = 1")
self.validate_identity(
"DECLARE @X INT, @Y VARCHAR(10)", "DECLARE @X AS INTEGER, @Y AS VARCHAR(10)"
)
self.validate_identity(
"declare @X int = (select col from table where id = 1)",
"DECLARE @X AS INTEGER = (SELECT col FROM table WHERE id = 1)",
)
self.validate_identity(
"declare @X TABLE (Id INT NOT NULL, Name VARCHAR(100) NOT NULL)",
"DECLARE @X AS TABLE (Id INTEGER NOT NULL, Name VARCHAR(100) NOT NULL)",
)
self.validate_identity(
"declare @X TABLE (Id INT NOT NULL, constraint PK_Id primary key (Id))",
"DECLARE @X AS TABLE (Id INTEGER NOT NULL, CONSTRAINT PK_Id PRIMARY KEY (Id))",
)
self.validate_identity(
"declare @X UserDefinedTableType",
"DECLARE @X AS UserDefinedTableType",
)
self.validate_identity(
"DECLARE @MyTableVar TABLE (EmpID INT NOT NULL, PRIMARY KEY CLUSTERED (EmpID), UNIQUE NONCLUSTERED (EmpID), INDEX CustomNonClusteredIndex NONCLUSTERED (EmpID))",
check_command_warning=True,
)
self.validate_identity(
"DECLARE vendor_cursor CURSOR FOR SELECT VendorID, Name FROM Purchasing.Vendor WHERE PreferredVendorStatus = 1 ORDER BY VendorID",
check_command_warning=True,
)

3
tests/fixtures/identity.sql vendored
View file

@ -869,4 +869,5 @@ TRUNCATE(a, b)
SELECT enum
SELECT unlogged
SELECT name
SELECT copy
SELECT copy
SELECT rollup

30
tests/fixtures/optimizer/tpc-ds/tpc-ds.sql vendored
View file

@ -1409,31 +1409,11 @@ WITH "_u_0" AS (
"store_sales"."ss_quantity" <= 80 AND "store_sales"."ss_quantity" >= 61
)
SELECT
CASE
WHEN MAX("_u_0"."_col_0") > 3672
THEN MAX("_u_1"."_col_0")
ELSE MAX("_u_2"."_col_0")
END AS "bucket1",
CASE
WHEN MAX("_u_3"."_col_0") > 3392
THEN MAX("_u_4"."_col_0")
ELSE MAX("_u_5"."_col_0")
END AS "bucket2",
CASE
WHEN MAX("_u_6"."_col_0") > 32784
THEN MAX("_u_7"."_col_0")
ELSE MAX("_u_8"."_col_0")
END AS "bucket3",
CASE
WHEN MAX("_u_9"."_col_0") > 26032
THEN MAX("_u_10"."_col_0")
ELSE MAX("_u_11"."_col_0")
END AS "bucket4",
CASE
WHEN MAX("_u_12"."_col_0") > 23982
THEN MAX("_u_13"."_col_0")
ELSE MAX("_u_14"."_col_0")
END AS "bucket5"
CASE WHEN "_u_0"."_col_0" > 3672 THEN "_u_1"."_col_0" ELSE "_u_2"."_col_0" END AS "bucket1",
CASE WHEN "_u_3"."_col_0" > 3392 THEN "_u_4"."_col_0" ELSE "_u_5"."_col_0" END AS "bucket2",
CASE WHEN "_u_6"."_col_0" > 32784 THEN "_u_7"."_col_0" ELSE "_u_8"."_col_0" END AS "bucket3",
CASE WHEN "_u_9"."_col_0" > 26032 THEN "_u_10"."_col_0" ELSE "_u_11"."_col_0" END AS "bucket4",
CASE WHEN "_u_12"."_col_0" > 23982 THEN "_u_13"."_col_0" ELSE "_u_14"."_col_0" END AS "bucket5"
FROM "reason" AS "reason"
CROSS JOIN "_u_0" AS "_u_0"
CROSS JOIN "_u_1" AS "_u_1"

312
tests/fixtures/optimizer/unnest_subqueries.sql vendored
View file

@ -1,243 +1,69 @@
--SELECT x.a > (SELECT SUM(y.a) AS b FROM y) FROM x;
--------------------------------------
-- Unnest Subqueries
--------------------------------------
SELECT *
FROM x AS x
WHERE
x.a = (SELECT SUM(y.a) AS a FROM y)
AND x.a IN (SELECT y.a AS a FROM y)
AND x.a IN (SELECT y.b AS b FROM y)
AND x.a = ANY (SELECT y.a AS a FROM y)
AND x.a = (SELECT SUM(y.b) AS b FROM y WHERE x.a = y.a)
AND x.a > (SELECT SUM(y.b) AS b FROM y WHERE x.a = y.a)
AND x.a <> ANY (SELECT y.a AS a FROM y WHERE y.a = x.a)
AND x.a NOT IN (SELECT y.a AS a FROM y WHERE y.a = x.a)
AND x.a IN (SELECT y.a AS a FROM y WHERE y.b = x.a)
AND x.a < (SELECT SUM(y.a) AS a FROM y WHERE y.a = x.a and y.a = x.b and y.b <> x.d)
AND EXISTS (SELECT y.a AS a, y.b AS b FROM y WHERE x.a = y.a)
AND x.a IN (SELECT y.a AS a FROM y LIMIT 10)
AND x.a IN (SELECT y.a AS a FROM y OFFSET 10)
AND x.a IN (SELECT y.a AS a, y.b AS b FROM y)
AND x.a > ANY (SELECT y.a FROM y)
AND x.a = (SELECT SUM(y.c) AS c FROM y WHERE y.a = x.a LIMIT 10)
AND x.a = (SELECT SUM(y.c) AS c FROM y WHERE y.a = x.a OFFSET 10)
AND x.a > ALL (SELECT y.c FROM y WHERE y.a = x.a)
AND x.a > (SELECT COUNT(*) as d FROM y WHERE y.a = x.a)
AND x.a = SUM(SELECT 1) -- invalid statement left alone
AND x.a IN (SELECT max(y.b) AS b FROM y GROUP BY y.a)
;
SELECT
*
FROM x AS x
CROSS JOIN (
SELECT
SUM(y.a) AS a
FROM y
) AS _u_0
LEFT JOIN (
SELECT
y.a AS a
FROM y
GROUP BY
y.a
) AS _u_1
ON x.a = _u_1.a
LEFT JOIN (
SELECT
y.b AS b
FROM y
GROUP BY
y.b
) AS _u_2
ON x.a = _u_2.b
LEFT JOIN (
SELECT
y.a AS a
FROM y
GROUP BY
y.a
) AS _u_3
ON x.a = _u_3.a
LEFT JOIN (
SELECT
SUM(y.b) AS b,
y.a AS _u_5
FROM y
WHERE
TRUE
GROUP BY
y.a
) AS _u_4
ON x.a = _u_4._u_5
LEFT JOIN (
SELECT
SUM(y.b) AS b,
y.a AS _u_7
FROM y
WHERE
TRUE
GROUP BY
y.a
) AS _u_6
ON x.a = _u_6._u_7
LEFT JOIN (
SELECT
y.a AS a
FROM y
WHERE
TRUE
GROUP BY
y.a
) AS _u_8
ON _u_8.a = x.a
LEFT JOIN (
SELECT
y.a AS a
FROM y
WHERE
TRUE
GROUP BY
y.a
) AS _u_9
ON _u_9.a = x.a
LEFT JOIN (
SELECT
ARRAY_AGG(y.a) AS a,
y.b AS _u_11
FROM y
WHERE
TRUE
GROUP BY
y.b
) AS _u_10
ON _u_10._u_11 = x.a
LEFT JOIN (
SELECT
SUM(y.a) AS a,
y.a AS _u_13,
ARRAY_AGG(y.b) AS _u_14
FROM y
WHERE
TRUE AND TRUE AND TRUE
GROUP BY
y.a
) AS _u_12
ON _u_12._u_13 = x.a AND _u_12._u_13 = x.b
LEFT JOIN (
SELECT
y.a AS a
FROM y
WHERE
TRUE
GROUP BY
y.a
) AS _u_15
ON x.a = _u_15.a
LEFT JOIN (
SELECT
ARRAY_AGG(c),
y.a AS _u_20
FROM y
WHERE
TRUE
GROUP BY
y.a
) AS _u_19
ON _u_19._u_20 = x.a
LEFT JOIN (
SELECT
COUNT(*) AS d,
y.a AS _u_22
FROM y
WHERE
TRUE
GROUP BY
y.a
) AS _u_21
ON _u_21._u_22 = x.a
LEFT JOIN (
SELECT
_q.b
FROM (
SELECT
MAX(y.b) AS b
FROM y
GROUP BY
y.a
) AS _q
GROUP BY
_q.b
) AS _u_24
ON x.a = _u_24.b
WHERE
x.a = _u_0.a
AND NOT _u_1.a IS NULL
AND NOT _u_2.b IS NULL
AND NOT _u_3.a IS NULL
AND x.a = _u_4.b
AND x.a > _u_6.b
AND x.a = _u_8.a
AND NOT x.a = _u_9.a
AND ARRAY_ANY(_u_10.a, _x -> _x = x.a)
AND (
x.a < _u_12.a AND ARRAY_ANY(_u_12._u_14, _x -> _x <> x.d)
)
AND NOT _u_15.a IS NULL
AND x.a IN (
SELECT
y.a AS a
FROM y
LIMIT 10
)
AND x.a IN (
SELECT
y.a AS a
FROM y
OFFSET 10
)
AND x.a IN (
SELECT
y.a AS a,
y.b AS b
FROM y
)
AND x.a > ANY (
SELECT
y.a
FROM y
)
AND x.a = (
SELECT
SUM(y.c) AS c
FROM y
WHERE
y.a = x.a
LIMIT 10
)
AND x.a = (
SELECT
SUM(y.c) AS c
FROM y
WHERE
y.a = x.a
OFFSET 10
)
AND ARRAY_ALL(_u_19."", _x -> _x = x.a)
AND x.a > COALESCE(_u_21.d, 0)
AND x.a = SUM(SELECT
1) /* invalid statement left alone */
AND NOT _u_24.b IS NULL
;
SELECT
CAST((
SELECT
x.a AS a
FROM x
) AS TEXT) AS a;
SELECT
CAST((
SELECT
x.a AS a
FROM x
) AS TEXT) AS a;
SELECT * FROM x WHERE x.a = (SELECT SUM(y.a) AS a FROM y);
SELECT * FROM x CROSS JOIN (SELECT SUM(y.a) AS a FROM y) AS _u_0 WHERE x.a = _u_0.a;
SELECT * FROM x WHERE x.a IN (SELECT y.a AS a FROM y);
SELECT * FROM x LEFT JOIN (SELECT y.a AS a FROM y GROUP BY y.a) AS _u_0 ON x.a = _u_0.a WHERE NOT _u_0.a IS NULL;
SELECT * FROM x WHERE x.a IN (SELECT y.b AS b FROM y);
SELECT * FROM x LEFT JOIN (SELECT y.b AS b FROM y GROUP BY y.b) AS _u_0 ON x.a = _u_0.b WHERE NOT _u_0.b IS NULL;
SELECT * FROM x WHERE x.a = ANY (SELECT y.a AS a FROM y);
SELECT * FROM x LEFT JOIN (SELECT y.a AS a FROM y GROUP BY y.a) AS _u_0 ON x.a = _u_0.a WHERE NOT _u_0.a IS NULL;
SELECT * FROM x WHERE x.a = (SELECT SUM(y.b) AS b FROM y WHERE x.a = y.a);
SELECT * FROM x LEFT JOIN (SELECT SUM(y.b) AS b, y.a AS _u_1 FROM y WHERE TRUE GROUP BY y.a) AS _u_0 ON x.a = _u_0._u_1 WHERE x.a = _u_0.b;
SELECT * FROM x WHERE x.a > (SELECT SUM(y.b) AS b FROM y WHERE x.a = y.a);
SELECT * FROM x LEFT JOIN (SELECT SUM(y.b) AS b, y.a AS _u_1 FROM y WHERE TRUE GROUP BY y.a) AS _u_0 ON x.a = _u_0._u_1 WHERE x.a > _u_0.b;
SELECT * FROM x WHERE x.a <> ANY (SELECT y.a AS a FROM y WHERE y.a = x.a);
SELECT * FROM x LEFT JOIN (SELECT y.a AS a FROM y WHERE TRUE GROUP BY y.a) AS _u_0 ON _u_0.a = x.a WHERE x.a <> _u_0.a;
SELECT * FROM x WHERE x.a NOT IN (SELECT y.a AS a FROM y WHERE y.a = x.a);
SELECT * FROM x LEFT JOIN (SELECT y.a AS a FROM y WHERE TRUE GROUP BY y.a) AS _u_0 ON _u_0.a = x.a WHERE NOT x.a = _u_0.a;
SELECT * FROM x WHERE x.a IN (SELECT y.a AS a FROM y WHERE y.b = x.a);
SELECT * FROM x LEFT JOIN (SELECT ARRAY_AGG(y.a) AS a, y.b AS _u_1 FROM y WHERE TRUE GROUP BY y.b) AS _u_0 ON _u_0._u_1 = x.a WHERE ARRAY_ANY(_u_0.a, _x -> _x = x.a);
SELECT * FROM x WHERE x.a < (SELECT SUM(y.a) AS a FROM y WHERE y.a = x.a and y.a = x.b and y.b <> x.d);
SELECT * FROM x LEFT JOIN (SELECT SUM(y.a) AS a, y.a AS _u_1, ARRAY_AGG(y.b) AS _u_2 FROM y WHERE TRUE AND TRUE AND TRUE GROUP BY y.a) AS _u_0 ON _u_0._u_1 = x.a AND _u_0._u_1 = x.b WHERE (x.a < _u_0.a AND ARRAY_ANY(_u_0._u_2, _x -> _x <> x.d));
SELECT * FROM x WHERE EXISTS (SELECT y.a AS a, y.b AS b FROM y WHERE x.a = y.a);
SELECT * FROM x LEFT JOIN (SELECT y.a AS a FROM y WHERE TRUE GROUP BY y.a) AS _u_0 ON x.a = _u_0.a WHERE NOT _u_0.a IS NULL;
SELECT * FROM x WHERE x.a IN (SELECT y.a AS a FROM y LIMIT 10);
SELECT * FROM x WHERE x.a IN (SELECT y.a AS a FROM y LIMIT 10);
SELECT * FROM x.a WHERE x.a IN (SELECT y.a AS a FROM y OFFSET 10);
SELECT * FROM x.a WHERE x.a IN (SELECT y.a AS a FROM y OFFSET 10);
SELECT * FROM x.a WHERE x.a IN (SELECT y.a AS a, y.b AS b FROM y);
SELECT * FROM x.a WHERE x.a IN (SELECT y.a AS a, y.b AS b FROM y);
SELECT * FROM x.a WHERE x.a > ANY (SELECT y.a FROM y);
SELECT * FROM x.a WHERE x.a > ANY (SELECT y.a FROM y);
SELECT * FROM x WHERE x.a = (SELECT SUM(y.c) AS c FROM y WHERE y.a = x.a LIMIT 10);
SELECT * FROM x WHERE x.a = (SELECT SUM(y.c) AS c FROM y WHERE y.a = x.a LIMIT 10);
SELECT * FROM x WHERE x.a = (SELECT SUM(y.c) AS c FROM y WHERE y.a = x.a OFFSET 10);
SELECT * FROM x WHERE x.a = (SELECT SUM(y.c) AS c FROM y WHERE y.a = x.a OFFSET 10);
SELECT * FROM x WHERE x.a > ALL (SELECT y.c AS c FROM y WHERE y.a = x.a);
SELECT * FROM x LEFT JOIN (SELECT ARRAY_AGG(y.c) AS c, y.a AS _u_1 FROM y WHERE TRUE GROUP BY y.a) AS _u_0 ON _u_0._u_1 = x.a WHERE ARRAY_ALL(_u_0.c, _x -> x.a > _x);
SELECT * FROM x WHERE x.a > (SELECT COUNT(*) as d FROM y WHERE y.a = x.a);
SELECT * FROM x LEFT JOIN (SELECT COUNT(*) AS d, y.a AS _u_1 FROM y WHERE TRUE GROUP BY y.a) AS _u_0 ON _u_0._u_1 = x.a WHERE x.a > COALESCE(_u_0.d, 0);
# title: invalid statement left alone
SELECT * FROM x WHERE x.a = SUM(SELECT 1);
SELECT * FROM x WHERE x.a = SUM(SELECT 1);
SELECT * FROM x WHERE x.a IN (SELECT max(y.b) AS b FROM y GROUP BY y.a);
SELECT * FROM x LEFT JOIN (SELECT _q.b AS b FROM (SELECT MAX(y.b) AS b FROM y GROUP BY y.a) AS _q GROUP BY _q.b) AS _u_0 ON x.a = _u_0.b WHERE NOT _u_0.b IS NULL;
SELECT x.a > (SELECT SUM(y.a) AS b FROM y) FROM x;
SELECT x.a > _u_0.b FROM x CROSS JOIN (SELECT SUM(y.a) AS b FROM y) AS _u_0;
SELECT (SELECT MAX(t2.c1) AS c1 FROM t2 WHERE t2.c2 = t1.c2 AND t2.c3 <= TRUNC(t1.c3)) AS c FROM t1;
SELECT _u_0.c1 AS c FROM t1 LEFT JOIN (SELECT MAX(t2.c1) AS c1, t2.c2 AS _u_1, MAX(t2.c3) AS _u_2 FROM t2 WHERE TRUE AND TRUE GROUP BY t2.c2) AS _u_0 ON _u_0._u_1 = t1.c2 WHERE _u_0._u_2 <= TRUNC(t1.c3);

10
tests/test_executor.py
View file

@ -707,9 +707,15 @@ class TestExecutor(unittest.TestCase):
("ROUND(1.2)", 1),
("ROUND(1.2345, 2)", 1.23),
("ROUND(NULL)", None),
("UNIXTOTIME(1659981729)", datetime.datetime(2022, 8, 8, 18, 2, 9)),
(
"UNIXTOTIME(1659981729)",
datetime.datetime(2022, 8, 8, 18, 2, 9, tzinfo=datetime.timezone.utc),
),
("TIMESTRTOTIME('2013-04-05 01:02:03')", datetime.datetime(2013, 4, 5, 1, 2, 3)),
("UNIXTOTIME(40 * 365 * 86400)", datetime.datetime(2009, 12, 22, 00, 00, 00)),
(
"UNIXTOTIME(40 * 365 * 86400)",
datetime.datetime(2009, 12, 22, 00, 00, 00, tzinfo=datetime.timezone.utc),
),
(
"STRTOTIME('08/03/2024 12:34:56', '%d/%m/%Y %H:%M:%S')",
datetime.datetime(2024, 3, 8, 12, 34, 56),

19
tests/test_expressions.py
View file

@ -674,7 +674,9 @@ class TestExpressions(unittest.TestCase):
self.assertIsInstance(parse_one("STANDARD_HASH('hello', 'sha256')"), exp.StandardHash)
self.assertIsInstance(parse_one("DATE(foo)"), exp.Date)
self.assertIsInstance(parse_one("HEX(foo)"), exp.Hex)
self.assertIsInstance(parse_one("TO_HEX(foo)", read="bigquery"), exp.Hex)
self.assertIsInstance(parse_one("LOWER(HEX(foo))"), exp.LowerHex)
self.assertIsInstance(parse_one("TO_HEX(foo)", read="bigquery"), exp.LowerHex)
self.assertIsInstance(parse_one("UPPER(TO_HEX(foo))", read="bigquery"), exp.Hex)
self.assertIsInstance(parse_one("TO_HEX(MD5(foo))", read="bigquery"), exp.MD5)
self.assertIsInstance(parse_one("TRANSFORM(a, b)", read="spark"), exp.Transform)
self.assertIsInstance(parse_one("ADD_MONTHS(a, b)"), exp.AddMonths)
@ -834,21 +836,22 @@ class TestExpressions(unittest.TestCase):
b AS B,
c, /*comment*/
d AS D, -- another comment
CAST(x AS INT) -- final comment
CAST(x AS INT), -- yet another comment
y AND /* foo */ w AS E -- final comment
FROM foo
"""
expression = parse_one(sql)
self.assertEqual(
[e.alias_or_name for e in expression.expressions],
["a", "B", "c", "D", "x"],
["a", "B", "c", "D", "x", "E"],
)
self.assertEqual(
expression.sql(),
"SELECT a, b AS B, c /* comment */, d AS D /* another comment */, CAST(x AS INT) /* final comment */ FROM foo",
"SELECT a, b AS B, c /* comment */, d AS D /* another comment */, CAST(x AS INT) /* yet another comment */, y AND /* foo */ w AS E /* final comment */ FROM foo",
)
self.assertEqual(
expression.sql(comments=False),
"SELECT a, b AS B, c, d AS D, CAST(x AS INT) FROM foo",
"SELECT a, b AS B, c, d AS D, CAST(x AS INT), y AND w AS E FROM foo",
)
self.assertEqual(
expression.sql(pretty=True, comments=False),
@ -857,7 +860,8 @@ class TestExpressions(unittest.TestCase):
b AS B,
c,
d AS D,
CAST(x AS INT)
CAST(x AS INT),
y AND w AS E
FROM foo""",
)
self.assertEqual(
@ -867,7 +871,8 @@ FROM foo""",
b AS B,
c, /* comment */
d AS D, /* another comment */
CAST(x AS INT) /* final comment */
CAST(x AS INT), /* yet another comment */
y AND /* foo */ w AS E /* final comment */
FROM foo""",
)

8
tests/test_lineage.py
View file

@ -487,3 +487,11 @@ class TestLineage(unittest.TestCase):
downstream = node.downstream[0]
self.assertEqual(downstream.name, "z.a")
self.assertEqual(downstream.source.sql(), "SELECT y.a AS a, y.b AS b, y.c AS c FROM y AS y")
def test_node_name_doesnt_contain_comment(self) -> None:
sql = "SELECT * FROM (SELECT x /* c */ FROM t1) AS t2"
node = lineage("x", sql)
self.assertEqual(len(node.downstream), 1)
self.assertEqual(len(node.downstream[0].downstream), 1)
self.assertEqual(node.downstream[0].downstream[0].name, "t1.x")

6
tests/test_optimizer.py
View file

@ -428,11 +428,7 @@ SELECT :with,WITH :expressions,CTE :this,UNION :this,SELECT :expressions,1,:expr
)
def test_unnest_subqueries(self):
self.check_file(
"unnest_subqueries",
optimizer.unnest_subqueries.unnest_subqueries,
pretty=True,
)
self.check_file("unnest_subqueries", optimizer.unnest_subqueries.unnest_subqueries)
def test_pushdown_predicates(self):
self.check_file("pushdown_predicates", optimizer.pushdown_predicates.pushdown_predicates)

35
tests/test_parser.py
View file

@ -503,7 +503,7 @@ class TestParser(unittest.TestCase):
self.assertIsInstance(set_item, exp.SetItem)
self.assertIsInstance(set_item.this, exp.EQ)
self.assertIsInstance(set_item.this.this, exp.Identifier)
self.assertIsInstance(set_item.this.this, exp.Column)
self.assertIsInstance(set_item.this.expression, exp.Literal)
self.assertEqual(set_item.args.get("kind"), "SESSION")
@ -856,5 +856,38 @@ class TestParser(unittest.TestCase):
with self.subTest(dialect):
self.assertEqual(parse_one(sql, dialect=dialect).sql(dialect=dialect), sql)
def test_alter_set(self):
sqls = [
"ALTER TABLE tbl SET TBLPROPERTIES ('x'='1', 'Z'='2')",
"ALTER TABLE tbl SET SERDE 'test' WITH SERDEPROPERTIES ('k'='v', 'kay'='vee')",
"ALTER TABLE tbl SET SERDEPROPERTIES ('k'='v', 'kay'='vee')",
"ALTER TABLE tbl SET LOCATION 'new_location'",
"ALTER TABLE tbl SET FILEFORMAT file_format",
"ALTER TABLE tbl SET TAGS ('tag1' = 't1', 'tag2' = 't2')",
]
for dialect in (
"hive",
"spark2",
"spark",
"databricks",
):
for sql in sqls:
with self.subTest(f"Testing query '{sql}' for dialect {dialect}"):
self.assertEqual(parse_one(sql, dialect=dialect).sql(dialect=dialect), sql)
def test_distinct_from(self):
self.assertIsInstance(parse_one("a IS DISTINCT FROM b OR c IS DISTINCT FROM d"), exp.Or)
def test_trailing_comments(self):
expressions = parse("""
select * from x;
-- my comment
""")
self.assertEqual(
";\n".join(e.sql() for e in expressions), "SELECT * FROM x;\n/* my comment */"
)
def test_parse_prop_eq(self):
self.assertIsInstance(parse_one("x(a := b and c)").expressions[0], exp.PropertyEQ)

28
tests/test_schema.py
View file

@ -271,6 +271,34 @@ class TestSchema(unittest.TestCase):
"Table z must match the schema's nesting level: 2.",
)
with self.assertRaises(SchemaError) as ctx:
MappingSchema(
{
"catalog": {
"db": {"tbl": {"col": "a"}},
},
"tbl2": {"col": "b"},
},
)
self.assertEqual(
str(ctx.exception),
"Table tbl2 must match the schema's nesting level: 3.",
)
with self.assertRaises(SchemaError) as ctx:
MappingSchema(
{
"tbl2": {"col": "b"},
"catalog": {
"db": {"tbl": {"col": "a"}},
},
},
)
self.assertEqual(
str(ctx.exception),
"Table catalog.db.tbl must match the schema's nesting level: 1.",
)
def test_has_column(self):
schema = MappingSchema({"x": {"c": "int"}})
self.assertTrue(schema.has_column("x", exp.column("c")))

7
tests/test_transpile.py
View file

@ -551,6 +551,13 @@ FROM x""",
pretty=True,
)
self.validate(
"""SELECT X FROM catalog.db.table WHERE Y
--
AND Z""",
"""SELECT X FROM catalog.db.table WHERE Y AND Z""",
)
def test_types(self):
self.validate("INT 1", "CAST(1 AS INT)")
self.validate("VARCHAR 'x' y", "CAST('x' AS VARCHAR) AS y")