Adding upstream version 0.5.0.
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
parent
f9051e9424
commit
16e40566d2
GPG key ID:
FBB4F0E80A80222F
8 changed files with 1303 additions and 0 deletions
29
LICENSE.md
Normal file
29
LICENSE.md
Normal file
|
|
@ -0,0 +1,29 @@
|
||||||
|
BSD 3-Clause License
|
||||||
|
|
||||||
|
Copyright (c) 2022-present, Gani Georgiev
|
||||||
|
All rights reserved.
|
||||||
|
|
||||||
|
Redistribution and use in source and binary forms, with or without
|
||||||
|
modification, are permitted provided that the following conditions are met:
|
||||||
|
|
||||||
|
1. Redistributions of source code must retain the above copyright notice, this
|
||||||
|
list of conditions and the following disclaimer.
|
||||||
|
|
||||||
|
2. Redistributions in binary form must reproduce the above copyright notice,
|
||||||
|
this list of conditions and the following disclaimer in the documentation
|
||||||
|
and/or other materials provided with the distribution.
|
||||||
|
|
||||||
|
3. Neither the name of the copyright holder nor the names of its
|
||||||
|
contributors may be used to endorse or promote products derived from
|
||||||
|
this software without specific prior written permission.
|
||||||
|
|
||||||
|
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
||||||
|
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||||
|
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
||||||
|
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
||||||
|
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||||
|
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
||||||
|
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||||
|
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
||||||
|
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||||
|
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||||
118
README.md
Normal file
118
README.md
Normal file
|
|
@ -0,0 +1,118 @@
|
||||||
|
fexpr
|
||||||
|
[](https://goreportcard.com/report/github.com/ganigeorgiev/fexpr)
|
||||||
|
[](https://pkg.go.dev/github.com/ganigeorgiev/fexpr)
|
||||||
|
================================================================================
|
||||||
|
|
||||||
|
**fexpr** is a filter query language parser that generates easy to work with AST structure so that you can create safely SQL, Elasticsearch, etc. queries from user input.
|
||||||
|
|
||||||
|
Or in other words, transform the string `"id > 1"` into the struct `[{&& {{identifier id} > {number 1}}}]`.
|
||||||
|
|
||||||
|
Supports parenthesis and various conditional expression operators (see [Grammar](https://github.com/ganigeorgiev/fexpr#grammar)).
|
||||||
|
|
||||||
|
|
||||||
|
## Example usage
|
||||||
|
|
||||||
|
```
|
||||||
|
go get github.com/ganigeorgiev/fexpr
|
||||||
|
```
|
||||||
|
|
||||||
|
```go
|
||||||
|
package main
|
||||||
|
|
||||||
|
import github.com/ganigeorgiev/fexpr
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
// [{&& {{identifier id} = {number 123}}} {&& {{identifier status} = {text active}}}]
|
||||||
|
result, err := fexpr.Parse("id=123 && status='active'")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
> Note that each parsed expression statement contains a join/union operator (`&&` or `||`) so that the result can be consumed on small chunks without having to rely on the group/nesting context.
|
||||||
|
|
||||||
|
> See the [package documentation](https://pkg.go.dev/github.com/ganigeorgiev/fexpr) for more details and examples.
|
||||||
|
|
||||||
|
|
||||||
|
## Grammar
|
||||||
|
|
||||||
|
**fexpr** grammar resembles the SQL `WHERE` expression syntax. It recognizes several token types (identifiers, numbers, quoted text, expression operators, whitespaces, etc.).
|
||||||
|
|
||||||
|
> You could find all supported tokens in [`scanner.go`](https://github.com/ganigeorgiev/fexpr/blob/master/scanner.go).
|
||||||
|
|
||||||
|
#### Operators
|
||||||
|
|
||||||
|
- **`=`** Equal operator (eg. `a=b`)
|
||||||
|
- **`!=`** NOT Equal operator (eg. `a!=b`)
|
||||||
|
- **`>`** Greater than operator (eg. `a>b`)
|
||||||
|
- **`>=`** Greater than or equal operator (eg. `a>=b`)
|
||||||
|
- **`<`** Less than or equal operator (eg. `a<b`)
|
||||||
|
- **`<=`** Less than or equal operator (eg. `a<=b`)
|
||||||
|
- **`~`** Like/Contains operator (eg. `a~b`)
|
||||||
|
- **`!~`** NOT Like/Contains operator (eg. `a!~b`)
|
||||||
|
- **`?=`** Array/Any equal operator (eg. `a?=b`)
|
||||||
|
- **`?!=`** Array/Any NOT Equal operator (eg. `a?!=b`)
|
||||||
|
- **`?>`** Array/Any Greater than operator (eg. `a?>b`)
|
||||||
|
- **`?>=`** Array/Any Greater than or equal operator (eg. `a?>=b`)
|
||||||
|
- **`?<`** Array/Any Less than or equal operator (eg. `a?<b`)
|
||||||
|
- **`?<=`** Array/Any Less than or equal operator (eg. `a?<=b`)
|
||||||
|
- **`?~`** Array/Any Like/Contains operator (eg. `a?~b`)
|
||||||
|
- **`?!~`** Array/Any NOT Like/Contains operator (eg. `a?!~b`)
|
||||||
|
- **`&&`** AND join operator (eg. `a=b && c=d`)
|
||||||
|
- **`||`** OR join operator (eg. `a=b || c=d`)
|
||||||
|
- **`()`** Parenthesis (eg. `(a=1 && b=2) || (a=3 && b=4)`)
|
||||||
|
|
||||||
|
#### Numbers
|
||||||
|
Number tokens are any integer or decimal numbers.
|
||||||
|
|
||||||
|
_Example_: `123`, `10.50`, `-14`.
|
||||||
|
|
||||||
|
#### Quoted text
|
||||||
|
|
||||||
|
Text tokens are any literals that are wrapped by `'` or `"` quotes.
|
||||||
|
|
||||||
|
_Example_: `'Lorem ipsum dolor 123!'`, `"escaped \"word\""`, `"mixed 'quotes' are fine"`.
|
||||||
|
|
||||||
|
#### Identifiers
|
||||||
|
|
||||||
|
Identifier tokens are literals that start with a letter, `_`, `@` or `#` and could contain further any number of letters, digits, `.` (usually used as a separator) or `:` (usually used as modifier) characters.
|
||||||
|
|
||||||
|
_Example_: `id`, `a.b.c`, `field123`, `@request.method`, `author.name:length`.
|
||||||
|
|
||||||
|
#### Functions
|
||||||
|
|
||||||
|
Function tokens are similar to the identifiers but in addition accept a list of arguments enclosed in parenthesis `()`.
|
||||||
|
The function arguments must be separated by comma (_a single trailing comma is also allowed_) and each argument can be an identifier, quoted text, number or another nested function (_up to 2 nested_).
|
||||||
|
|
||||||
|
_Example_: `test()`, `test(a.b, 123, "abc")`, `@a.b.c:test(true)`, `a(b(c(1, 2)))`.
|
||||||
|
|
||||||
|
#### Comments
|
||||||
|
|
||||||
|
Comment tokens are any single line text literals starting with `//`.
|
||||||
|
Similar to whitespaces, comments are ignored by `fexpr.Parse()`.
|
||||||
|
|
||||||
|
_Example_: `// test`.
|
||||||
|
|
||||||
|
|
||||||
|
## Using only the scanner
|
||||||
|
|
||||||
|
The tokenizer (aka. `fexpr.Scanner`) could be used without the parser's state machine so that you can write your own custom tokens processing:
|
||||||
|
|
||||||
|
```go
|
||||||
|
s := fexpr.NewScanner([]byte("id > 123"))
|
||||||
|
|
||||||
|
// scan single token at a time until EOF or error is reached
|
||||||
|
for {
|
||||||
|
t, err := s.Scan()
|
||||||
|
if t.Type == fexpr.TokenEOF || err != nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Println(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Output:
|
||||||
|
// {<nil> identifier id}
|
||||||
|
// {<nil> whitespace }
|
||||||
|
// {<nil> sign >}
|
||||||
|
// {<nil> whitespace }
|
||||||
|
// {<nil> number 123}
|
||||||
|
```
|
||||||
36
examples_test.go
Normal file
36
examples_test.go
Normal file
|
|
@ -0,0 +1,36 @@
|
||||||
|
package fexpr_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/ganigeorgiev/fexpr"
|
||||||
|
)
|
||||||
|
|
||||||
|
func ExampleScanner_Scan() {
|
||||||
|
s := fexpr.NewScanner([]byte("id > 123"))
|
||||||
|
|
||||||
|
for {
|
||||||
|
t, err := s.Scan()
|
||||||
|
if t.Type == fexpr.TokenEOF || err != nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Println(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Output:
|
||||||
|
// {<nil> identifier id}
|
||||||
|
// {<nil> whitespace }
|
||||||
|
// {<nil> sign >}
|
||||||
|
// {<nil> whitespace }
|
||||||
|
// {<nil> number 123}
|
||||||
|
}
|
||||||
|
|
||||||
|
func ExampleParse() {
|
||||||
|
result, _ := fexpr.Parse("id > 123")
|
||||||
|
|
||||||
|
fmt.Println(result)
|
||||||
|
|
||||||
|
// Output:
|
||||||
|
// [{{{<nil> identifier id} > {<nil> number 123}} &&}]
|
||||||
|
}
|
||||||
3
go.mod
Normal file
3
go.mod
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
module github.com/ganigeorgiev/fexpr
|
||||||
|
|
||||||
|
go 1.16
|
||||||
130
parser.go
Normal file
130
parser.go
Normal file
|
|
@ -0,0 +1,130 @@
|
||||||
|
package fexpr
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
)
|
||||||
|
|
||||||
|
var ErrEmpty = errors.New("empty filter expression")
|
||||||
|
var ErrIncomplete = errors.New("invalid or incomplete filter expression")
|
||||||
|
var ErrInvalidComment = errors.New("invalid comment")
|
||||||
|
|
||||||
|
// Expr represents an individual tokenized expression consisting
|
||||||
|
// of left operand, operator and a right operand.
|
||||||
|
type Expr struct {
|
||||||
|
Left Token
|
||||||
|
Op SignOp
|
||||||
|
Right Token
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsZero checks if the current Expr has zero-valued props.
|
||||||
|
func (e Expr) IsZero() bool {
|
||||||
|
return e.Op == "" && e.Left.Literal == "" && e.Left.Type == "" && e.Right.Literal == "" && e.Right.Type == ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExprGroup represents a wrapped expression and its join type.
|
||||||
|
//
|
||||||
|
// The group's Item could be either an `Expr` instance or `[]ExprGroup` slice (for nested expressions).
|
||||||
|
type ExprGroup struct {
|
||||||
|
Item interface{}
|
||||||
|
Join JoinOp
|
||||||
|
}
|
||||||
|
|
||||||
|
// parser's state machine steps
|
||||||
|
const (
|
||||||
|
stepBeforeSign = iota
|
||||||
|
stepSign
|
||||||
|
stepAfterSign
|
||||||
|
StepJoin
|
||||||
|
)
|
||||||
|
|
||||||
|
// Parse parses the provided text and returns its processed AST
|
||||||
|
// in the form of `ExprGroup` slice(s).
|
||||||
|
//
|
||||||
|
// Comments and whitespaces are ignored.
|
||||||
|
func Parse(text string) ([]ExprGroup, error) {
|
||||||
|
result := []ExprGroup{}
|
||||||
|
scanner := NewScanner([]byte(text))
|
||||||
|
step := stepBeforeSign
|
||||||
|
join := JoinAnd
|
||||||
|
|
||||||
|
var expr Expr
|
||||||
|
|
||||||
|
for {
|
||||||
|
t, err := scanner.Scan()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if t.Type == TokenEOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
if t.Type == TokenWS || t.Type == TokenComment {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if t.Type == TokenGroup {
|
||||||
|
groupResult, err := Parse(t.Literal)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// append only if non-empty group
|
||||||
|
if len(groupResult) > 0 {
|
||||||
|
result = append(result, ExprGroup{Join: join, Item: groupResult})
|
||||||
|
}
|
||||||
|
|
||||||
|
step = StepJoin
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
switch step {
|
||||||
|
case stepBeforeSign:
|
||||||
|
if t.Type != TokenIdentifier && t.Type != TokenText && t.Type != TokenNumber && t.Type != TokenFunction {
|
||||||
|
return nil, fmt.Errorf("expected left operand (identifier, function, text or number), got %q (%s)", t.Literal, t.Type)
|
||||||
|
}
|
||||||
|
|
||||||
|
expr = Expr{Left: t}
|
||||||
|
|
||||||
|
step = stepSign
|
||||||
|
case stepSign:
|
||||||
|
if t.Type != TokenSign {
|
||||||
|
return nil, fmt.Errorf("expected a sign operator, got %q (%s)", t.Literal, t.Type)
|
||||||
|
}
|
||||||
|
|
||||||
|
expr.Op = SignOp(t.Literal)
|
||||||
|
step = stepAfterSign
|
||||||
|
case stepAfterSign:
|
||||||
|
if t.Type != TokenIdentifier && t.Type != TokenText && t.Type != TokenNumber && t.Type != TokenFunction {
|
||||||
|
return nil, fmt.Errorf("expected right operand (identifier, function text or number), got %q (%s)", t.Literal, t.Type)
|
||||||
|
}
|
||||||
|
|
||||||
|
expr.Right = t
|
||||||
|
result = append(result, ExprGroup{Join: join, Item: expr})
|
||||||
|
|
||||||
|
step = StepJoin
|
||||||
|
case StepJoin:
|
||||||
|
if t.Type != TokenJoin {
|
||||||
|
return nil, fmt.Errorf("expected && or ||, got %q (%s)", t.Literal, t.Type)
|
||||||
|
}
|
||||||
|
|
||||||
|
join = JoinAnd
|
||||||
|
if t.Literal == "||" {
|
||||||
|
join = JoinOr
|
||||||
|
}
|
||||||
|
|
||||||
|
step = stepBeforeSign
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if step != StepJoin {
|
||||||
|
if len(result) == 0 && expr.IsZero() {
|
||||||
|
return nil, ErrEmpty
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, ErrIncomplete
|
||||||
|
}
|
||||||
|
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
142
parser_test.go
Normal file
142
parser_test.go
Normal file
|
|
@ -0,0 +1,142 @@
|
||||||
|
package fexpr
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExprIzZero(t *testing.T) {
|
||||||
|
scenarios := []struct {
|
||||||
|
expr Expr
|
||||||
|
result bool
|
||||||
|
}{
|
||||||
|
{Expr{}, true},
|
||||||
|
{Expr{Op: SignAnyEq}, false},
|
||||||
|
{Expr{Left: Token{Literal: "123"}}, false},
|
||||||
|
{Expr{Left: Token{Type: TokenWS}}, false},
|
||||||
|
{Expr{Right: Token{Literal: "123"}}, false},
|
||||||
|
{Expr{Right: Token{Type: TokenWS}}, false},
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, s := range scenarios {
|
||||||
|
t.Run(fmt.Sprintf("s%d", i), func(t *testing.T) {
|
||||||
|
if v := s.expr.IsZero(); v != s.result {
|
||||||
|
t.Fatalf("Expected %v, got %v for \n%v", s.result, v, s.expr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParse(t *testing.T) {
|
||||||
|
scenarios := []struct {
|
||||||
|
input string
|
||||||
|
expectedError bool
|
||||||
|
expectedPrint string
|
||||||
|
}{
|
||||||
|
{`> 1`, true, "[]"},
|
||||||
|
{`a >`, true, "[]"},
|
||||||
|
{`a > >`, true, "[]"},
|
||||||
|
{`a > %`, true, "[]"},
|
||||||
|
{`a ! 1`, true, "[]"},
|
||||||
|
{`a - 1`, true, "[]"},
|
||||||
|
{`a + 1`, true, "[]"},
|
||||||
|
{`1 - 1`, true, "[]"},
|
||||||
|
{`1 + 1`, true, "[]"},
|
||||||
|
{`> a 1`, true, "[]"},
|
||||||
|
{`a || 1`, true, "[]"},
|
||||||
|
{`a && 1`, true, "[]"},
|
||||||
|
{`test > 1 &&`, true, `[]`},
|
||||||
|
{`|| test = 1`, true, `[]`},
|
||||||
|
{`test = 1 && ||`, true, "[]"},
|
||||||
|
{`test = 1 && a`, true, "[]"},
|
||||||
|
{`test = 1 && a`, true, "[]"},
|
||||||
|
{`test = 1 && "a"`, true, "[]"},
|
||||||
|
{`test = 1 a`, true, "[]"},
|
||||||
|
{`test = 1 a`, true, "[]"},
|
||||||
|
{`test = 1 "a"`, true, "[]"},
|
||||||
|
{`test = 1@test`, true, "[]"},
|
||||||
|
{`test = .@test`, true, "[]"},
|
||||||
|
// mismatched text quotes
|
||||||
|
{`test = "demo'`, true, "[]"},
|
||||||
|
{`test = 'demo"`, true, "[]"},
|
||||||
|
{`test = 'demo'"`, true, "[]"},
|
||||||
|
{`test = 'demo''`, true, "[]"},
|
||||||
|
{`test = "demo"'`, true, "[]"},
|
||||||
|
{`test = "demo""`, true, "[]"},
|
||||||
|
{`test = ""demo""`, true, "[]"},
|
||||||
|
{`test = ''demo''`, true, "[]"},
|
||||||
|
{"test = `demo`", true, "[]"},
|
||||||
|
// comments
|
||||||
|
{"test = / demo", true, "[]"},
|
||||||
|
{"test = // demo", true, "[]"},
|
||||||
|
{"// demo", true, "[]"},
|
||||||
|
{"test = 123 // demo", false, "[{{{<nil> identifier test} = {<nil> number 123}} &&}]"},
|
||||||
|
{"test = // demo\n123", false, "[{{{<nil> identifier test} = {<nil> number 123}} &&}]"},
|
||||||
|
{`
|
||||||
|
a = 123 &&
|
||||||
|
// demo
|
||||||
|
b = 456
|
||||||
|
`, false, "[{{{<nil> identifier a} = {<nil> number 123}} &&} {{{<nil> identifier b} = {<nil> number 456}} &&}]"},
|
||||||
|
// functions
|
||||||
|
{`test() = 12`, false, `[{{{[] function test} = {<nil> number 12}} &&}]`},
|
||||||
|
{`(a.b.c(1) = d.e.f(2)) || 1=2`, false, `[{[{{{[{<nil> number 1}] function a.b.c} = {[{<nil> number 2}] function d.e.f}} &&}] &&} {{{<nil> number 1} = {<nil> number 2}} ||}]`},
|
||||||
|
// valid simple expression and sign operators check
|
||||||
|
{`1=12`, false, `[{{{<nil> number 1} = {<nil> number 12}} &&}]`},
|
||||||
|
{` 1 = 12 `, false, `[{{{<nil> number 1} = {<nil> number 12}} &&}]`},
|
||||||
|
{`"demo" != test`, false, `[{{{<nil> text demo} != {<nil> identifier test}} &&}]`},
|
||||||
|
{`a~1`, false, `[{{{<nil> identifier a} ~ {<nil> number 1}} &&}]`},
|
||||||
|
{`a !~ 1`, false, `[{{{<nil> identifier a} !~ {<nil> number 1}} &&}]`},
|
||||||
|
{`test>12`, false, `[{{{<nil> identifier test} > {<nil> number 12}} &&}]`},
|
||||||
|
{`test > 12`, false, `[{{{<nil> identifier test} > {<nil> number 12}} &&}]`},
|
||||||
|
{`test >="test"`, false, `[{{{<nil> identifier test} >= {<nil> text test}} &&}]`},
|
||||||
|
{`test<@demo.test2`, false, `[{{{<nil> identifier test} < {<nil> identifier @demo.test2}} &&}]`},
|
||||||
|
{`1<="test"`, false, `[{{{<nil> number 1} <= {<nil> text test}} &&}]`},
|
||||||
|
{`1<="te'st"`, false, `[{{{<nil> number 1} <= {<nil> text te'st}} &&}]`},
|
||||||
|
{`demo='te\'st'`, false, `[{{{<nil> identifier demo} = {<nil> text te'st}} &&}]`},
|
||||||
|
{`demo="te\'st"`, false, `[{{{<nil> identifier demo} = {<nil> text te\'st}} &&}]`},
|
||||||
|
{`demo="te\"st"`, false, `[{{{<nil> identifier demo} = {<nil> text te"st}} &&}]`},
|
||||||
|
// invalid parenthesis
|
||||||
|
{`(a=1`, true, `[]`},
|
||||||
|
{`a=1)`, true, `[]`},
|
||||||
|
{`((a=1)`, true, `[]`},
|
||||||
|
{`{a=1}`, true, `[]`},
|
||||||
|
{`[a=1]`, true, `[]`},
|
||||||
|
{`((a=1 || a=2) && c=1))`, true, `[]`},
|
||||||
|
// valid parenthesis
|
||||||
|
{`()`, true, `[]`},
|
||||||
|
{`(a=1)`, false, `[{[{{{<nil> identifier a} = {<nil> number 1}} &&}] &&}]`},
|
||||||
|
{`(a="test(")`, false, `[{[{{{<nil> identifier a} = {<nil> text test(}} &&}] &&}]`},
|
||||||
|
{`(a="test)")`, false, `[{[{{{<nil> identifier a} = {<nil> text test)}} &&}] &&}]`},
|
||||||
|
{`((a=1))`, false, `[{[{[{{{<nil> identifier a} = {<nil> number 1}} &&}] &&}] &&}]`},
|
||||||
|
{`a=1 || 2!=3`, false, `[{{{<nil> identifier a} = {<nil> number 1}} &&} {{{<nil> number 2} != {<nil> number 3}} ||}]`},
|
||||||
|
{`a=1 && 2!=3`, false, `[{{{<nil> identifier a} = {<nil> number 1}} &&} {{{<nil> number 2} != {<nil> number 3}} &&}]`},
|
||||||
|
{`a=1 && 2!=3 || "b"=a`, false, `[{{{<nil> identifier a} = {<nil> number 1}} &&} {{{<nil> number 2} != {<nil> number 3}} &&} {{{<nil> text b} = {<nil> identifier a}} ||}]`},
|
||||||
|
{`(a=1 && 2!=3) || "b"=a`, false, `[{[{{{<nil> identifier a} = {<nil> number 1}} &&} {{{<nil> number 2} != {<nil> number 3}} &&}] &&} {{{<nil> text b} = {<nil> identifier a}} ||}]`},
|
||||||
|
{`((a=1 || a=2) && (c=1))`, false, `[{[{[{{{<nil> identifier a} = {<nil> number 1}} &&} {{{<nil> identifier a} = {<nil> number 2}} ||}] &&} {[{{{<nil> identifier c} = {<nil> number 1}} &&}] &&}] &&}]`},
|
||||||
|
// https://github.com/pocketbase/pocketbase/issues/5017
|
||||||
|
{`(a='"')`, false, `[{[{{{<nil> identifier a} = {<nil> text "}} &&}] &&}]`},
|
||||||
|
{`(a='\'')`, false, `[{[{{{<nil> identifier a} = {<nil> text '}} &&}] &&}]`},
|
||||||
|
{`(a="'")`, false, `[{[{{{<nil> identifier a} = {<nil> text '}} &&}] &&}]`},
|
||||||
|
{`(a="\"")`, false, `[{[{{{<nil> identifier a} = {<nil> text "}} &&}] &&}]`},
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, scenario := range scenarios {
|
||||||
|
t.Run(fmt.Sprintf("s%d:%s", i, scenario.input), func(t *testing.T) {
|
||||||
|
v, err := Parse(scenario.input)
|
||||||
|
|
||||||
|
if scenario.expectedError && err == nil {
|
||||||
|
t.Fatalf("Expected error, got nil (%q)", scenario.input)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !scenario.expectedError && err != nil {
|
||||||
|
t.Fatalf("Did not expect error, got %q (%q).", err, scenario.input)
|
||||||
|
}
|
||||||
|
|
||||||
|
vPrint := fmt.Sprintf("%v", v)
|
||||||
|
|
||||||
|
if vPrint != scenario.expectedPrint {
|
||||||
|
t.Fatalf("Expected %s, got %s", scenario.expectedPrint, vPrint)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
679
scanner.go
Normal file
679
scanner.go
Normal file
|
|
@ -0,0 +1,679 @@
|
||||||
|
package fexpr
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"unicode/utf8"
|
||||||
|
)
|
||||||
|
|
||||||
|
// eof represents a marker rune for the end of the reader.
|
||||||
|
const eof = rune(0)
|
||||||
|
|
||||||
|
// JoinOp represents a join type operator.
|
||||||
|
type JoinOp string
|
||||||
|
|
||||||
|
// supported join type operators
|
||||||
|
const (
|
||||||
|
JoinAnd JoinOp = "&&"
|
||||||
|
JoinOr JoinOp = "||"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SignOp represents an expression sign operator.
|
||||||
|
type SignOp string
|
||||||
|
|
||||||
|
// supported expression sign operators
|
||||||
|
const (
|
||||||
|
SignEq SignOp = "="
|
||||||
|
SignNeq SignOp = "!="
|
||||||
|
SignLike SignOp = "~"
|
||||||
|
SignNlike SignOp = "!~"
|
||||||
|
SignLt SignOp = "<"
|
||||||
|
SignLte SignOp = "<="
|
||||||
|
SignGt SignOp = ">"
|
||||||
|
SignGte SignOp = ">="
|
||||||
|
|
||||||
|
// array/any operators
|
||||||
|
SignAnyEq SignOp = "?="
|
||||||
|
SignAnyNeq SignOp = "?!="
|
||||||
|
SignAnyLike SignOp = "?~"
|
||||||
|
SignAnyNlike SignOp = "?!~"
|
||||||
|
SignAnyLt SignOp = "?<"
|
||||||
|
SignAnyLte SignOp = "?<="
|
||||||
|
SignAnyGt SignOp = "?>"
|
||||||
|
SignAnyGte SignOp = "?>="
|
||||||
|
)
|
||||||
|
|
||||||
|
// TokenType represents a Token type.
|
||||||
|
type TokenType string
|
||||||
|
|
||||||
|
// token type constants
|
||||||
|
const (
|
||||||
|
TokenUnexpected TokenType = "unexpected"
|
||||||
|
TokenEOF TokenType = "eof"
|
||||||
|
TokenWS TokenType = "whitespace"
|
||||||
|
TokenJoin TokenType = "join"
|
||||||
|
TokenSign TokenType = "sign"
|
||||||
|
TokenIdentifier TokenType = "identifier" // variable, column name, placeholder, etc.
|
||||||
|
TokenFunction TokenType = "function" // function
|
||||||
|
TokenNumber TokenType = "number"
|
||||||
|
TokenText TokenType = "text" // ' or " quoted string
|
||||||
|
TokenGroup TokenType = "group" // groupped/nested tokens
|
||||||
|
TokenComment TokenType = "comment"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Token represents a single scanned literal (one or more combined runes).
|
||||||
|
type Token struct {
|
||||||
|
Meta interface{}
|
||||||
|
Type TokenType
|
||||||
|
Literal string
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewScanner creates and returns a new scanner instance loaded with the specified data.
|
||||||
|
func NewScanner(data []byte) *Scanner {
|
||||||
|
return &Scanner{
|
||||||
|
data: data,
|
||||||
|
maxFuncDepth: 3,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scanner represents a filter and lexical scanner.
|
||||||
|
type Scanner struct {
|
||||||
|
data []byte
|
||||||
|
pos int
|
||||||
|
maxFuncDepth int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scan reads and returns the next available token value from the scanner's buffer.
|
||||||
|
func (s *Scanner) Scan() (Token, error) {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
return Token{Type: TokenEOF, Literal: ""}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if isWhitespaceRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
return s.scanWhitespace()
|
||||||
|
}
|
||||||
|
|
||||||
|
if isGroupStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
return s.scanGroup()
|
||||||
|
}
|
||||||
|
|
||||||
|
if isIdentifierStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
return s.scanIdentifier(s.maxFuncDepth)
|
||||||
|
}
|
||||||
|
|
||||||
|
if isNumberStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
return s.scanNumber()
|
||||||
|
}
|
||||||
|
|
||||||
|
if isTextStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
return s.scanText(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
if isSignStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
return s.scanSign()
|
||||||
|
}
|
||||||
|
|
||||||
|
if isJoinStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
return s.scanJoin()
|
||||||
|
}
|
||||||
|
|
||||||
|
if isCommentStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
return s.scanComment()
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenUnexpected, Literal: string(ch)}, fmt.Errorf("unexpected character %q", ch)
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanWhitespace consumes all contiguous whitespace runes.
|
||||||
|
func (s *Scanner) scanWhitespace() (Token, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
// Reads every subsequent whitespace character into the buffer.
|
||||||
|
// Non-whitespace runes and EOF will cause the loop to exit.
|
||||||
|
for {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
if !isWhitespaceRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// write the whitespace rune
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenWS, Literal: buf.String()}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanNumber consumes all contiguous digit runes
|
||||||
|
// (complex numbers and scientific notations are not supported).
|
||||||
|
func (s *Scanner) scanNumber() (Token, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
var hadDot bool
|
||||||
|
|
||||||
|
// Read every subsequent digit rune into the buffer.
|
||||||
|
// Non-digit runes and EOF will cause the loop to exit.
|
||||||
|
for {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// not a digit rune
|
||||||
|
if !isDigitRune(ch) &&
|
||||||
|
// minus sign but not at the beginning
|
||||||
|
(ch != '-' || buf.Len() != 0) &&
|
||||||
|
// dot but there was already another dot
|
||||||
|
(ch != '.' || hadDot) {
|
||||||
|
s.unread()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// write the rune
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
|
||||||
|
if ch == '.' {
|
||||||
|
hadDot = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
total := buf.Len()
|
||||||
|
literal := buf.String()
|
||||||
|
|
||||||
|
var err error
|
||||||
|
// only "-" or starts with "." or ends with "."
|
||||||
|
if (total == 1 && literal[0] == '-') || literal[0] == '.' || literal[total-1] == '.' {
|
||||||
|
err = fmt.Errorf("invalid number %q", literal)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenNumber, Literal: buf.String()}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanText consumes all contiguous quoted text runes.
|
||||||
|
func (s *Scanner) scanText(preserveQuotes bool) (Token, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
// read the first rune to determine the quotes type
|
||||||
|
firstCh := s.read()
|
||||||
|
buf.WriteRune(firstCh)
|
||||||
|
var prevCh rune
|
||||||
|
var hasMatchingQuotes bool
|
||||||
|
|
||||||
|
// Read every subsequent text rune into the buffer.
|
||||||
|
// EOF and matching unescaped ending quote will cause the loop to exit.
|
||||||
|
for {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// write the text rune
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
|
||||||
|
// unescaped matching quote, aka. the end
|
||||||
|
if ch == firstCh && prevCh != '\\' {
|
||||||
|
hasMatchingQuotes = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
prevCh = ch
|
||||||
|
}
|
||||||
|
|
||||||
|
literal := buf.String()
|
||||||
|
|
||||||
|
var err error
|
||||||
|
if !hasMatchingQuotes {
|
||||||
|
err = fmt.Errorf("invalid quoted text %q", literal)
|
||||||
|
} else if !preserveQuotes {
|
||||||
|
// unquote
|
||||||
|
literal = literal[1 : len(literal)-1]
|
||||||
|
// remove escaped quotes prefix (aka. \)
|
||||||
|
firstChStr := string(firstCh)
|
||||||
|
literal = strings.ReplaceAll(literal, `\`+firstChStr, firstChStr)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenText, Literal: literal}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanComment consumes all contiguous single line comment runes until
|
||||||
|
// a new character (\n) or EOF is reached.
|
||||||
|
func (s *Scanner) scanComment() (Token, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
// Read the first 2 characters without writting them to the buffer.
|
||||||
|
if !isCommentStartRune(s.read()) || !isCommentStartRune(s.read()) {
|
||||||
|
return Token{Type: TokenComment}, ErrInvalidComment
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read every subsequent comment text rune into the buffer.
|
||||||
|
// \n and EOF will cause the loop to exit.
|
||||||
|
for i := 0; ; i++ {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof || ch == '\n' {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenComment, Literal: strings.TrimSpace(buf.String())}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanIdentifier consumes all contiguous ident runes.
|
||||||
|
func (s *Scanner) scanIdentifier(funcDepth int) (Token, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
// read the first rune in case it is a special start identifier character
|
||||||
|
buf.WriteRune(s.read())
|
||||||
|
|
||||||
|
// Read every subsequent identifier rune into the buffer.
|
||||||
|
// Non-ident runes and EOF will cause the loop to exit.
|
||||||
|
for {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// func
|
||||||
|
if ch == '(' {
|
||||||
|
funcName := buf.String()
|
||||||
|
if funcDepth <= 0 {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName}, fmt.Errorf("max nested function arguments reached (max: %d)", s.maxFuncDepth)
|
||||||
|
}
|
||||||
|
if !isValidIdentifier(funcName) {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName}, fmt.Errorf("invalid function name %q", funcName)
|
||||||
|
}
|
||||||
|
s.unread()
|
||||||
|
return s.scanFunctionArgs(funcName, funcDepth)
|
||||||
|
}
|
||||||
|
|
||||||
|
// not an identifier character
|
||||||
|
if !isLetterRune(ch) && !isDigitRune(ch) && !isIdentifierCombineRune(ch) && ch != '_' {
|
||||||
|
s.unread()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// write the identifier rune
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
}
|
||||||
|
|
||||||
|
literal := buf.String()
|
||||||
|
|
||||||
|
var err error
|
||||||
|
if !isValidIdentifier(literal) {
|
||||||
|
err = fmt.Errorf("invalid identifier %q", literal)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenIdentifier, Literal: literal}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanSign consumes all contiguous sign operator runes.
|
||||||
|
func (s *Scanner) scanSign() (Token, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
// Read every subsequent sign rune into the buffer.
|
||||||
|
// Non-sign runes and EOF will cause the loop to exit.
|
||||||
|
for {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
if !isSignStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// write the sign rune
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
}
|
||||||
|
|
||||||
|
literal := buf.String()
|
||||||
|
|
||||||
|
var err error
|
||||||
|
if !isSignOperator(literal) {
|
||||||
|
err = fmt.Errorf("invalid sign operator %q", literal)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenSign, Literal: literal}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanJoin consumes all contiguous join operator runes.
|
||||||
|
func (s *Scanner) scanJoin() (Token, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
// Read every subsequent join operator rune into the buffer.
|
||||||
|
// Non-join runes and EOF will cause the loop to exit.
|
||||||
|
for {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
if !isJoinStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// write the join operator rune
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
}
|
||||||
|
|
||||||
|
literal := buf.String()
|
||||||
|
|
||||||
|
var err error
|
||||||
|
if !isJoinOperator(literal) {
|
||||||
|
err = fmt.Errorf("invalid join operator %q", literal)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenJoin, Literal: literal}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanGroup consumes all runes within a group/parenthesis.
|
||||||
|
func (s *Scanner) scanGroup() (Token, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
// read the first group bracket without writing it to the buffer
|
||||||
|
firstChar := s.read()
|
||||||
|
openGroups := 1
|
||||||
|
|
||||||
|
// Read every subsequent text rune into the buffer.
|
||||||
|
// EOF and matching unescaped ending quote will cause the loop to exit.
|
||||||
|
for {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
if isGroupStartRune(ch) {
|
||||||
|
// nested group
|
||||||
|
openGroups++
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
} else if isTextStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
t, err := s.scanText(true) // with quotes to preserve the exact text start/end runes
|
||||||
|
if err != nil {
|
||||||
|
// write the errored literal as it is
|
||||||
|
buf.WriteString(t.Literal)
|
||||||
|
return Token{Type: TokenGroup, Literal: buf.String()}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
buf.WriteString(t.Literal)
|
||||||
|
} else if ch == ')' {
|
||||||
|
openGroups--
|
||||||
|
|
||||||
|
if openGroups <= 0 {
|
||||||
|
// main group end
|
||||||
|
break
|
||||||
|
} else {
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
buf.WriteRune(ch)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
literal := buf.String()
|
||||||
|
|
||||||
|
var err error
|
||||||
|
if !isGroupStartRune(firstChar) || openGroups > 0 {
|
||||||
|
err = fmt.Errorf("invalid formatted group - missing %d closing bracket(s)", openGroups)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenGroup, Literal: literal}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanFunctionArgs consumes all contiguous function call runes to
|
||||||
|
// extract its arguments and returns a function token with the found
|
||||||
|
// Token arguments loaded in Token.Meta.
|
||||||
|
func (s *Scanner) scanFunctionArgs(funcName string, funcDepth int) (Token, error) {
|
||||||
|
var args []Token
|
||||||
|
|
||||||
|
var expectComma, isComma, isClosed bool
|
||||||
|
|
||||||
|
ch := s.read()
|
||||||
|
if ch != '(' {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName}, fmt.Errorf("invalid or incomplete function call %q", funcName)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read every subsequent rune until ')' or EOF has been reached.
|
||||||
|
for {
|
||||||
|
ch := s.read()
|
||||||
|
|
||||||
|
if ch == eof {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
if ch == ')' {
|
||||||
|
isClosed = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// skip whitespaces
|
||||||
|
if isWhitespaceRune(ch) {
|
||||||
|
_, err := s.scanWhitespace()
|
||||||
|
if err != nil {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("failed to scan whitespaces in function %q: %w", funcName, err)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// skip comments
|
||||||
|
if isCommentStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
_, err := s.scanComment()
|
||||||
|
if err != nil {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("failed to scan comment in function %q: %w", funcName, err)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
isComma = ch == ','
|
||||||
|
|
||||||
|
if expectComma && !isComma {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("expected comma after the last argument in function %q", funcName)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !expectComma && isComma {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("unexpected comma in function %q", funcName)
|
||||||
|
}
|
||||||
|
|
||||||
|
expectComma = false // reset
|
||||||
|
|
||||||
|
if isComma {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if isIdentifierStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
t, err := s.scanIdentifier(funcDepth - 1)
|
||||||
|
if err != nil {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("invalid identifier argument %q in function %q: %w", t.Literal, funcName, err)
|
||||||
|
}
|
||||||
|
args = append(args, t)
|
||||||
|
expectComma = true
|
||||||
|
} else if isNumberStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
t, err := s.scanNumber()
|
||||||
|
if err != nil {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("invalid number argument %q in function %q: %w", t.Literal, funcName, err)
|
||||||
|
}
|
||||||
|
args = append(args, t)
|
||||||
|
expectComma = true
|
||||||
|
} else if isTextStartRune(ch) {
|
||||||
|
s.unread()
|
||||||
|
t, err := s.scanText(false)
|
||||||
|
if err != nil {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("invalid text argument %q in function %q: %w", t.Literal, funcName, err)
|
||||||
|
}
|
||||||
|
args = append(args, t)
|
||||||
|
expectComma = true
|
||||||
|
} else {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("unsupported argument character %q in function %q", ch, funcName)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !isClosed {
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, fmt.Errorf("invalid or incomplete function %q (expected ')')", funcName)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Token{Type: TokenFunction, Literal: funcName, Meta: args}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// unread unreads the last character and revert the position 1 step back.
|
||||||
|
func (s *Scanner) unread() {
|
||||||
|
if s.pos > 0 {
|
||||||
|
s.pos = s.pos - 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// read reads the next rune and moves the position forward.
|
||||||
|
func (s *Scanner) read() rune {
|
||||||
|
if s.pos >= len(s.data) {
|
||||||
|
return eof
|
||||||
|
}
|
||||||
|
|
||||||
|
ch, n := utf8.DecodeRune(s.data[s.pos:])
|
||||||
|
s.pos += n
|
||||||
|
|
||||||
|
return ch
|
||||||
|
}
|
||||||
|
|
||||||
|
// Lexical helpers:
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
// isWhitespaceRune checks if a rune is a space, tab, or newline.
|
||||||
|
func isWhitespaceRune(ch rune) bool { return ch == ' ' || ch == '\t' || ch == '\n' }
|
||||||
|
|
||||||
|
// isLetterRune checks if a rune is a letter.
|
||||||
|
func isLetterRune(ch rune) bool {
|
||||||
|
return (ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z')
|
||||||
|
}
|
||||||
|
|
||||||
|
// isDigitRune checks if a rune is a digit.
|
||||||
|
func isDigitRune(ch rune) bool {
|
||||||
|
return (ch >= '0' && ch <= '9')
|
||||||
|
}
|
||||||
|
|
||||||
|
// isTextStartRune checks if a rune is a valid quoted text first character
|
||||||
|
// (aka. single or double quote).
|
||||||
|
func isTextStartRune(ch rune) bool {
|
||||||
|
return ch == '\'' || ch == '"'
|
||||||
|
}
|
||||||
|
|
||||||
|
// isNumberStartRune checks if a rune is a valid number start character (aka. digit).
|
||||||
|
func isNumberStartRune(ch rune) bool {
|
||||||
|
return ch == '-' || isDigitRune(ch)
|
||||||
|
}
|
||||||
|
|
||||||
|
// isSignStartRune checks if a rune is a valid sign operator start character.
|
||||||
|
func isSignStartRune(ch rune) bool {
|
||||||
|
return ch == '=' ||
|
||||||
|
ch == '?' ||
|
||||||
|
ch == '!' ||
|
||||||
|
ch == '>' ||
|
||||||
|
ch == '<' ||
|
||||||
|
ch == '~'
|
||||||
|
}
|
||||||
|
|
||||||
|
// isJoinStartRune checks if a rune is a valid join type start character.
|
||||||
|
func isJoinStartRune(ch rune) bool {
|
||||||
|
return ch == '&' || ch == '|'
|
||||||
|
}
|
||||||
|
|
||||||
|
// isGroupStartRune checks if a rune is a valid group/parenthesis start character.
|
||||||
|
func isGroupStartRune(ch rune) bool {
|
||||||
|
return ch == '('
|
||||||
|
}
|
||||||
|
|
||||||
|
// isCommentStartRune checks if a rune is a valid comment start character.
|
||||||
|
func isCommentStartRune(ch rune) bool {
|
||||||
|
return ch == '/'
|
||||||
|
}
|
||||||
|
|
||||||
|
// isIdentifierStartRune checks if a rune is valid identifier's first character.
|
||||||
|
func isIdentifierStartRune(ch rune) bool {
|
||||||
|
return isLetterRune(ch) || isIdentifierSpecialStartRune(ch)
|
||||||
|
}
|
||||||
|
|
||||||
|
// isIdentifierSpecialStartRune checks if a rune is valid identifier's first special character.
|
||||||
|
func isIdentifierSpecialStartRune(ch rune) bool {
|
||||||
|
return ch == '@' || ch == '_' || ch == '#'
|
||||||
|
}
|
||||||
|
|
||||||
|
// isIdentifierCombineRune checks if a rune is valid identifier's combine character.
|
||||||
|
func isIdentifierCombineRune(ch rune) bool {
|
||||||
|
return ch == '.' || ch == ':'
|
||||||
|
}
|
||||||
|
|
||||||
|
// isSignOperator checks if a literal is a valid sign operator.
|
||||||
|
func isSignOperator(literal string) bool {
|
||||||
|
switch SignOp(literal) {
|
||||||
|
case
|
||||||
|
SignEq,
|
||||||
|
SignNeq,
|
||||||
|
SignLt,
|
||||||
|
SignLte,
|
||||||
|
SignGt,
|
||||||
|
SignGte,
|
||||||
|
SignLike,
|
||||||
|
SignNlike,
|
||||||
|
SignAnyEq,
|
||||||
|
SignAnyNeq,
|
||||||
|
SignAnyLike,
|
||||||
|
SignAnyNlike,
|
||||||
|
SignAnyLt,
|
||||||
|
SignAnyLte,
|
||||||
|
SignAnyGt,
|
||||||
|
SignAnyGte:
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// isJoinOperator checks if a literal is a valid join type operator.
|
||||||
|
func isJoinOperator(literal string) bool {
|
||||||
|
switch JoinOp(literal) {
|
||||||
|
case
|
||||||
|
JoinAnd,
|
||||||
|
JoinOr:
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// isValidIdentifier validates the literal against common identifier requirements.
|
||||||
|
func isValidIdentifier(literal string) bool {
|
||||||
|
length := len(literal)
|
||||||
|
|
||||||
|
return (
|
||||||
|
// doesn't end with combine rune
|
||||||
|
!isIdentifierCombineRune(rune(literal[length-1])) &&
|
||||||
|
// is not just a special start rune
|
||||||
|
(length != 1 || !isIdentifierSpecialStartRune(rune(literal[0]))))
|
||||||
|
}
|
||||||
166
scanner_test.go
Normal file
166
scanner_test.go
Normal file
|
|
@ -0,0 +1,166 @@
|
||||||
|
package fexpr
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNewScanner(t *testing.T) {
|
||||||
|
s := NewScanner([]byte("test"))
|
||||||
|
|
||||||
|
data := string(s.data)
|
||||||
|
|
||||||
|
if data != "test" {
|
||||||
|
t.Errorf("Expected the scanner reader data to be %q, got %q", "test", data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestScannerScan(t *testing.T) {
|
||||||
|
type output struct {
|
||||||
|
error bool
|
||||||
|
print string
|
||||||
|
}
|
||||||
|
testScenarios := []struct {
|
||||||
|
text string
|
||||||
|
expects []output
|
||||||
|
}{
|
||||||
|
// whitespace
|
||||||
|
{" ", []output{{false, "{<nil> whitespace }"}}},
|
||||||
|
{"test 123", []output{{false, "{<nil> identifier test}"}, {false, "{<nil> whitespace }"}, {false, "{<nil> number 123}"}}},
|
||||||
|
// identifier
|
||||||
|
{`test`, []output{{false, `{<nil> identifier test}`}}},
|
||||||
|
{`@`, []output{{true, `{<nil> identifier @}`}}},
|
||||||
|
{`test:`, []output{{true, `{<nil> identifier test:}`}}},
|
||||||
|
{`test.`, []output{{true, `{<nil> identifier test.}`}}},
|
||||||
|
{`@test.123:c`, []output{{false, `{<nil> identifier @test.123:c}`}}},
|
||||||
|
{`_test_a.123`, []output{{false, `{<nil> identifier _test_a.123}`}}},
|
||||||
|
{`#test.123:456`, []output{{false, `{<nil> identifier #test.123:456}`}}},
|
||||||
|
{`.test.123`, []output{{true, `{<nil> unexpected .}`}, {false, `{<nil> identifier test.123}`}}},
|
||||||
|
{`:test.123`, []output{{true, `{<nil> unexpected :}`}, {false, `{<nil> identifier test.123}`}}},
|
||||||
|
{`test#@`, []output{{false, `{<nil> identifier test}`}, {true, `{<nil> identifier #}`}, {true, `{<nil> identifier @}`}}},
|
||||||
|
{`test'`, []output{{false, `{<nil> identifier test}`}, {true, `{<nil> text '}`}}},
|
||||||
|
{`test"d`, []output{{false, `{<nil> identifier test}`}, {true, `{<nil> text "d}`}}},
|
||||||
|
// number
|
||||||
|
{`123`, []output{{false, `{<nil> number 123}`}}},
|
||||||
|
{`-123`, []output{{false, `{<nil> number -123}`}}},
|
||||||
|
{`-123.456`, []output{{false, `{<nil> number -123.456}`}}},
|
||||||
|
{`123.456`, []output{{false, `{<nil> number 123.456}`}}},
|
||||||
|
{`12.34.56`, []output{{false, `{<nil> number 12.34}`}, {true, `{<nil> unexpected .}`}, {false, `{<nil> number 56}`}}},
|
||||||
|
{`.123`, []output{{true, `{<nil> unexpected .}`}, {false, `{<nil> number 123}`}}},
|
||||||
|
{`- 123`, []output{{true, `{<nil> number -}`}, {false, `{<nil> whitespace }`}, {false, `{<nil> number 123}`}}},
|
||||||
|
{`12-3`, []output{{false, `{<nil> number 12}`}, {false, `{<nil> number -3}`}}},
|
||||||
|
{`123.abc`, []output{{true, `{<nil> number 123.}`}, {false, `{<nil> identifier abc}`}}},
|
||||||
|
// text
|
||||||
|
{`""`, []output{{false, `{<nil> text }`}}},
|
||||||
|
{`''`, []output{{false, `{<nil> text }`}}},
|
||||||
|
{`'test'`, []output{{false, `{<nil> text test}`}}},
|
||||||
|
{`'te\'st'`, []output{{false, `{<nil> text te'st}`}}},
|
||||||
|
{`"te\"st"`, []output{{false, `{<nil> text te"st}`}}},
|
||||||
|
{`"tes@#,;!@#%^'\"t"`, []output{{false, `{<nil> text tes@#,;!@#%^'"t}`}}},
|
||||||
|
{`'tes@#,;!@#%^\'"t'`, []output{{false, `{<nil> text tes@#,;!@#%^'"t}`}}},
|
||||||
|
{`"test`, []output{{true, `{<nil> text "test}`}}},
|
||||||
|
{`'test`, []output{{true, `{<nil> text 'test}`}}},
|
||||||
|
{`'АБЦ`, []output{{true, `{<nil> text 'АБЦ}`}}},
|
||||||
|
// join types
|
||||||
|
{`&&||`, []output{{true, `{<nil> join &&||}`}}},
|
||||||
|
{`&& ||`, []output{{false, `{<nil> join &&}`}, {false, `{<nil> whitespace }`}, {false, `{<nil> join ||}`}}},
|
||||||
|
{`'||test&&'&&123`, []output{{false, `{<nil> text ||test&&}`}, {false, `{<nil> join &&}`}, {false, `{<nil> number 123}`}}},
|
||||||
|
// expression signs
|
||||||
|
{`=!=`, []output{{true, `{<nil> sign =!=}`}}},
|
||||||
|
{`= != ~ !~ > >= < <= ?= ?!= ?~ ?!~ ?> ?>= ?< ?<=`, []output{
|
||||||
|
{false, `{<nil> sign =}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign !=}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ~}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign !~}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign >}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign >=}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign <}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign <=}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ?=}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ?!=}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ?~}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ?!~}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ?>}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ?>=}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ?<}`},
|
||||||
|
{false, `{<nil> whitespace }`},
|
||||||
|
{false, `{<nil> sign ?<=}`},
|
||||||
|
}},
|
||||||
|
// comments
|
||||||
|
{`/ test`, []output{{true, `{<nil> comment }`}, {false, `{<nil> identifier test}`}}},
|
||||||
|
{`/ / test`, []output{{true, `{<nil> comment }`}, {true, `{<nil> comment }`}, {false, `{<nil> identifier test}`}}},
|
||||||
|
{`//`, []output{{false, `{<nil> comment }`}}},
|
||||||
|
{`//test`, []output{{false, `{<nil> comment test}`}}},
|
||||||
|
{`// test`, []output{{false, `{<nil> comment test}`}}},
|
||||||
|
{`// test1 //test2 `, []output{{false, `{<nil> comment test1 //test2}`}}},
|
||||||
|
{`///test`, []output{{false, `{<nil> comment /test}`}}},
|
||||||
|
// funcs
|
||||||
|
{`test()`, []output{{false, `{[] function test}`}}},
|
||||||
|
{`test(a, b`, []output{{true, `{[{<nil> identifier a} {<nil> identifier b}] function test}`}}},
|
||||||
|
{`@test:abc()`, []output{{false, `{[] function @test:abc}`}}},
|
||||||
|
{`test( a )`, []output{{false, `{[{<nil> identifier a}] function test}`}}}, // with whitespaces
|
||||||
|
{`test(a, b)`, []output{{false, `{[{<nil> identifier a} {<nil> identifier b}] function test}`}}},
|
||||||
|
{`test(a, b, )`, []output{{false, `{[{<nil> identifier a} {<nil> identifier b}] function test}`}}}, // single trailing comma
|
||||||
|
{`test(a,,)`, []output{{true, `{[{<nil> identifier a}] function test}`}, {true, `{<nil> unexpected )}`}}}, // unexpected trailing commas
|
||||||
|
{`test(a,,,b)`, []output{{true, `{[{<nil> identifier a}] function test}`}, {true, `{<nil> unexpected ,}`}, {false, `{<nil> identifier b}`}, {true, `{<nil> unexpected )}`}}}, // unexpected mid-args commas
|
||||||
|
{`test( @test.a.b:test , 123, "ab)c", 'd,ce', false)`, []output{{false, `{[{<nil> identifier @test.a.b:test} {<nil> number 123} {<nil> text ab)c} {<nil> text d,ce} {<nil> identifier false}] function test}`}}},
|
||||||
|
{"test(a //test)", []output{{true, `{[{<nil> identifier a}] function test}`}}}, // invalid simple comment
|
||||||
|
{"test(a //test\n)", []output{{false, `{[{<nil> identifier a}] function test}`}}}, // valid simple comment
|
||||||
|
{"test(a, //test\n, b)", []output{{true, `{[{<nil> identifier a}] function test}`}, {false, `{<nil> whitespace }`}, {false, `{<nil> identifier b}`}, {true, `{<nil> unexpected )}`}}},
|
||||||
|
{"test(a, //test\n b)", []output{{false, `{[{<nil> identifier a} {<nil> identifier b}] function test}`}}},
|
||||||
|
{"test(a, test(test(b), c), d)", []output{{false, `{[{<nil> identifier a} {[{[{<nil> identifier b}] function test} {<nil> identifier c}] function test} {<nil> identifier d}] function test}`}}},
|
||||||
|
// max funcs depth
|
||||||
|
{"a(b(c(1)))", []output{{false, `{[{[{[{<nil> number 1}] function c}] function b}] function a}`}}},
|
||||||
|
{"a(b(c(d(1))))", []output{{true, `{[] function a}`}, {false, `{<nil> number 1}`}, {true, `{<nil> unexpected )}`}, {true, `{<nil> unexpected )}`}, {true, `{<nil> unexpected )}`}, {true, `{<nil> unexpected )}`}}},
|
||||||
|
// groups/parenthesis
|
||||||
|
{`a)`, []output{{false, `{<nil> identifier a}`}, {true, `{<nil> unexpected )}`}}},
|
||||||
|
{`(a b c`, []output{{true, `{<nil> group a b c}`}}},
|
||||||
|
{`(a b c)`, []output{{false, `{<nil> group a b c}`}}},
|
||||||
|
{`((a b c))`, []output{{false, `{<nil> group (a b c)}`}}},
|
||||||
|
{`((a )b c))`, []output{{false, `{<nil> group (a )b c}`}, {true, `{<nil> unexpected )}`}}},
|
||||||
|
{`("ab)("c)`, []output{{false, `{<nil> group "ab)("c}`}}},
|
||||||
|
{`("ab)(c)`, []output{{true, `{<nil> group "ab)(c)}`}}},
|
||||||
|
{`( func(1, 2, 3, func(4)) a b c )`, []output{{false, `{<nil> group func(1, 2, 3, func(4)) a b c }`}}},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, scenario := range testScenarios {
|
||||||
|
t.Run(scenario.text, func(t *testing.T) {
|
||||||
|
s := NewScanner([]byte(scenario.text))
|
||||||
|
|
||||||
|
// scan the text tokens
|
||||||
|
for j, expect := range scenario.expects {
|
||||||
|
token, err := s.Scan()
|
||||||
|
|
||||||
|
hasErr := err != nil
|
||||||
|
if expect.error != hasErr {
|
||||||
|
t.Errorf("[%d] Expected hasErr %v, got %v: %v (%v)", j, expect.error, hasErr, err, token)
|
||||||
|
}
|
||||||
|
|
||||||
|
tokenPrint := fmt.Sprintf("%v", token)
|
||||||
|
if tokenPrint != expect.print {
|
||||||
|
t.Errorf("[%d] Expected token %s, got %s", j, expect.print, tokenPrint)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// the last remaining token should be the eof
|
||||||
|
lastToken, err := s.Scan()
|
||||||
|
if err != nil || lastToken.Type != TokenEOF {
|
||||||
|
t.Fatalf("Expected EOF token, got %v (%v)", lastToken, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
Loading…
Add table
Add a link
Reference in a new issue