python/pysilfont
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.8.0.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-10 05:17:32 +01:00
parent c48d95b7fa
commit e40b3259c1
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
2403 changed files with 153656 additions and 0 deletions

106
docs/composite.md Normal file
View file

@ -0,0 +1,106 @@
# Defining composite glyphs
A composite glyph is one that is defined in terms of one or more other glyphs.
The composite definition syntax described in this document is a subset of the [GlyphConstruction](https://github.com/typemytype/GlyphConstruction) syntax used by Robofont, but with extensions for additional functionality.
Composites defined in this syntax can be applied to a UFO using the [psfbuildcomp](scripts.md#psfbuildcomp) tool.
# Overview
Each composite definition is on a single line and has the format:
```
<result> = <one or more glyphs> <parameters> # comment
```
where
- `<result>` is the name of the composite glyph being constructed
- `<one or more glyphs>` represents one or more glyphs used in the construction of the composite glyph, with optional glyph-level parameters described below
- `<parameters>` represents adjustments made to the `<result>` glyph, using the following optional parameters:
- at most one of the two following options:
- `^x,y` (where `x` is the amount added to the left margin and `y` is the amount added to the right margin)
- `^a` (where `a` is the advance width of the resulting glyph)
- `|usv` where `usv` is the 4-, 5- or 6-digit hex Unicode scalar value assigned to the resulting glyph
- `!colordef` (currently ignored by SIL tools)
- `[key1=value1;key2=value2;...]` to add one or more `key=value` pairs (representing SIL-specific properties documented below) to the resulting glyph
- `# comment` is an optional comment (everything from the `#` to the end of the line is ignored)
In addition, a line that begins with `#` is considered a comment and is ignored (as are blank lines).
If `[key=value]` properties for the resulting glyph are specified but no `|usv` is specified, then a `|` must be included before the `[`.
This insures that the properties are applied to the resulting composite glyph and not to the last glyph in the composite specification.
# Examples
In the following examples,
- `glyph` represents the resulting glyph being constructed
- `base`, `base1`, `base2` represent base glyphs being used in the construction
- `diac`, `diac1`, `diac2` represent diacritic glyphs being used in the construction
- `AP` represents an attachment point (also known as an anchor)
## glyph = base
```
Minus = Hyphen
```
This defines one glyph (`Minus`) in terms of another (`Hyphen`), without having to duplicate the contours used to create the shape.
## glyph = base1 & base2
```
ffi = f & f & i
```
This construct causes a glyph to be composed by aligning the origin of each successive base with the origin+advancewidth of the previous base. Unless overridden by the `^` parameter, the left sidebearing of the composite is that of the first base, the right sidebearing is that of the last, and the advancewidth of the composite is the sum of the advance widths of all the component base glyphs. [Unsure how this changes for right-to-left scripts]
## glyph = base + diac@AP
```
Aacute = A + acute@U
```
The resulting composite has the APs of the base(s), minus any APs used to attach the diacritics, plus the APs of the diacritics (adjusted for any displacement, as would be the case for stacked diacritics). In this example, glyph `acute` attaches to glyph `A` at AP `U` on `A` (by default using the `_U` AP on `tilde`). The `U` AP from `A` is removed (as is the `_U` AP on the `tilde`) and the `U` AP from `acute` is added.
Unless overridden by the `^` parameter, the advance width of the resulting composite is that of the base.
## glyph = base + diac1@AP + diac2@APonpreviousdiac
```
Ocircumflexacute = O + circumflex@U + acute@U
```
The acute is positioned according to the `U` AP on the immediately preceding glyph (`circumflex`), not the `U` AP on the base (`O`).
## glyph = base + diac@anyglyph:anyAP
The syntax allows you to express diacritic positioning using any arbitrary AP on any arbitrary glyph in the font, for example:
```
barredOacute = barredO + acute@O:U # not supported
```
Current SIL tools, however, only support an `anyglyph` that appears earlier in the composite definition, so the above example is **not** supported.
This syntax, however, makes it possible to override the default behavior of attaching to the immediately preceding glyph, so the following is supported (since the `@O:L` refers to the glyph `O` which appears earlier in the definition):
```
Ocircumflexdotaccent = O + circumflex@U + dotaccent@O:L
```
The `@O:L` causes the `dotaccent` diacritic to attach to the base glyph `O` (rather the immediately preceding `circumflex` glyph) using the `L` AP on the glyph `O` and the `_L` AP on the glyph `dotaccent`.
## glyph = base + diac@AP | usv
```
Aacute = A + acute@U | 00C1
```
USV is always given as four- to six-digit hexadecimal number with no leading "U+" or "0x".
## glyph = base + diac@AP ^ leftmarginadd,rightmarginadd
```
itilde = i + tilde@U ^ 50,50
```
This adds the values (in design units) to the left and right sidebearings. Note that these values could be negative.
# SIL Extensions
SIL extensions are all expressed as property lists (`key=value`) separated by semicolons and enclosed in square brackets: `[key1=value1;key2=value2]`.
- Properties that apply to a glyph being used in the construction of the composite appear after the glyph.
- Properties that apply to the resulting composite glyph appear after `|` (either that of the `|usv` or a single `|` if no `|usv` is present).
## glyph = base + diac@atAP[with=AP]
```
Aacute = A + acute@Ucap[with=_U]
```
The `with` property can be used to override the default AP, \_AP convention. The `_U` attachment point on the `acute` glyph is paired with the `Ucap` attachment point on the
## glyph = base + diac@AP[shift=x,y]
Aacute = A + acute@U[shift=100,100]
By applying the `shift` property to the `acute` glyph, the position of the diacritic relative to the base glyph `A` is changed.

204
docs/docs.md Normal file
View file

@ -0,0 +1,204 @@
# Pysilfont - utilities for font development
Pysilfont is a collection of tools to support font development, with an emphasis on [UFO](#ufo-support-in-pysilfont)-based workflows.
In addition to the UFO utilities, there is also support for testing using [FTML](#font-test-markup-language) and [Composite Definitions](#composite-definitions).
Some scripts are written specifically to fit in with the approaches recommended in [Font Development Best Practices](https://silnrsi.github.io/FDBP/en-US/index.html)
# Documentation
Documentation is held in the following documents:
- docs.md: This document - the main document for users
- [scripts.md](scripts.md): User documentation for all command-line tools and other scripts
- [technical.md](technical.md): Technical details for those wanting write scripts or other development tasks
- Other sub-documents, with links from the above
Installation instructions are in [README.md](../README.md)
# Scripts and commands
Many Pysilfont scripts are installed to be used as command-line tools, and these are all listed, with usage instructions, in [scripts.md](scripts.md). This also has details of some other example python scripts.
All scripts work using a standard framework designed to give users a consistent interface across scripts, and common features of these scripts are described in the following sections, so the **documentation below** needs to be read in conjunction with that in [scripts.md](scripts.md).
## Standard command line options
Nearly all scripts support these:
- `-h, --help`
- Basic usage help for the command
- `-h d`
- Display -h info with added info about default values
- `-h p`
- Display information about parameters (-p --params below)
- `-q, --quiet`
- Quiet mode - only display severe errors. See reporting below
- `-l LOG, --log LOG`
- Log file name (if not using the default name). By default logs will go in a logs subdirectory. If just a directory path is given, the log will go in there using the default name.
- `-p PARAMS, --params PARAMS`
- Other parameters - see below
The individual script documentation in scripts.md should indicate if some don't apply for a particular script
(There is also a hidden option --nq which overrides -q for use with automated systems like [smith](https://github.com/silnrsi/smith) which run scripts using -q by default)
# Parameters
There are many parameters that can be set to change the behaviour of scripts, either on the command line (using -p) or via a config file.
To set a parameter on the command line, use ``-p <param name>=<param value>``, eg
```
psfnormalize font.ufo -p scrlevel=w
```
-p can be used multiple times on a single command.
Commonly used command-line parameters include:
- scrlevel, loglevel
- Set the screen/logfile level from increasingly verbose options
- E - Errors
- P - Progress (default for scrlevel)
- W - Warnings (default for loglevel)
- I - Information
- V - Verbose
- checkfix (UFOs only)
- Validity tests when opening UFOs. Choice of None, Check, Fix with default Check
- See description of check & fix under [normalization](#normalization)
For a full list of parameters and how to set them via a config file (or in a UFO font) see [parameters.md](parameters.md).
## Default values
Most scripts have defaults for file names and other arguments - except for the main file the script is running against.
### Font/file name defaults
Once the initial input file (eg input font) has been given, most other font and file names will have defaults based on those.
This applies to other input font names, output font names, input file names and output file names and is done to minimise retyping repeated information like the path the files reside in. For example, simply using:
```
psfsetpsnames path/font.ufo
```
will:
- open (and update) `path/font.ufo`
- backup the font to `path/backups/font.ufo.nnn~`
- read its input from `path/font_psnames.csv`
- write its log to `path/logs/font_psnames.log`
If only part of a file name is supplied, other parts will default. So if only "test" is supplied for the output font name, the font would be output to `path/test.ufo`.
If a full file name is supplied, but no path, the current working directory will be used, so if “test.ufo” is supplied it won’t have `path/` added.
### Other defaults
Other parameters will just have standard default values.
### Displaying defaults for a command
Use `-h d` to see what defaults are for a given command. For example,
```
psfsetpsnames -h d
```
will output its help text with the following appended:
```
Defaults for parameters/options
Font/file names
-i _PSnames.csv
-l _PSnames.log
```
If the default value starts with “\_” (as with \_PSnames.csv above) then the input file name will be prepended to the default value; otherwise just the default value will be used.
## Reporting
Most scripts support standardised reporting (logging), both to screen and a log file, with different levels of reporting available. Levels are set for via loglevel and scrlevel parameters which can be set to one of:
- E Errors
- P Progress - Reports basic progress messages and all errors
- W Warning - As P but with warning messages as well
- I Info - As W but with information messages as well
- V Verbose - even more messages!
For most scripts these default to W for loglevel and P for scrlevel and can be set using -p (eg to set screen reporting to verbose use -p scrlevel=v).
-q --quiet sets quiet mode where all normal screen messages are suppressed. However, if there are any errors during the script execution, a single message is output on completion listing the counts for errors and warnings.
## Backups for fonts
If the output font name is the same as the input font name (which is the default behaviour for most scripts), then a backup is made original font prior to updating it.
By default, the last 5 copies of backups are kept in a sub-directory called “backups”. These defaults can be changed using the following parameters:
- `backup` - if set to 0, no backups are done
- `backupdir` - alternative directory for backups
- `backupkeep` - number of backups to keep
# UFO support in Pysilfont
With some limitations, all UFO scripts in Pysilfont should work with UFO2 or UFO3 source files - and can convert from one format to the other.
In addition most scripts will output in a normalized form, designed to work with source control systems. Most aspects of the normalization can be set by parameters, so projects are not forced to use Pysilfont’s default normalization.
The simplest script is psfnormalize, which will normalize a UFO (and optionally convert between UFO 2 and UFO3 if -v is used to specify the alternative version)
Note that other scripts also normalize, so psfnormalize is usually only needed after fonts have been processed by external font tools.
## Normalization
By default scripts normalize the UFOs and also run various check & fix tests to ensure the validity of the UFO metadata.
Default normalization behaviours include:
- XML formatting
- Use 2 spaces as indents
- Don’t indent the ``<dict>`` for plists
- Sort all ``<dict>``s in ascending key order
- Where values can be “integer or float”, store integer values as ``<integer>``
- Limit ``<real>`` limit decimal precision to 6
- For attributes identified as numeric, limit decimal precision to 6
- glif file names - use the UFO 3 suggested algorithm, even for UFO 2 fonts
- order glif elements and attributes in the order they are described in the UFO spec
Most of the above can be overridden by [parameters](#parameters)
The check & fix tests are based on [Font Development Best Practices](https://silnrsi.github.io/FDBP/en-US/index.html) and include:
- fontinfo.plist
- Required fields
- Fields to be deleted
- Fields to constructed from other fields
- Specific recommended values for some fields
- lib.plist
- Required fields
- Recommended values
- Fields that should not be present
The check & fix behaviour can be controlled by [parameters](#parameters), currently just the checkfix parameter which defaults to 'check' (just report what is wrong), but can be set to 'fix' to fix what it can, or none for no checking.
## Known limitations
The following are known limitations that will be addressed in the future:
- UFO 3 specific folders (data and images) are preserved, even if present in a UFO 2 font.
- Converting from UFO 3 to UFO 2 only handles data that has a place in UFO 2, but does include converting UFO 3 anchors to the standard way of handling them in UFO 2
- If a project uses non-standard files within the UFO folder, they are deleted
# Font Test Markup Language
Font Test Markup Language (FTML) is a file format for specifying the content and structure of font test data. It is designed to support complex test data, such as strings with specific language tags or data that should presented with certain font features activated. It also allows for indication of what portions of test data are in focus and which are only present to provide context.
FTML is described in the [FTML github project](https://github.com/silnrsi/ftml).
Pysilfont includes some python scripts for working with FTML, and a python library, [ftml.py](technical.md#ftml.py), so that new scripts can be developed to read and write FTML files.
# Composite definitions
Pysilfont includes tools for automatically adding composite glyphs to fonts. The syntax used for composite definitions is a subset of that used by RoboFont plus some extensions - see [Composite Tools](https://silnrsi.github.io/FDBP/en-US/Composite_Tools.html) in the Font Development Best Practices documentation for more details.
The current tools (psfbuildcomp, psfcomp2xml and psfxml2comp) are documented in [scripts.md](scripts.md).
The tools are based on a python module, [comp.py](technical.md#comppy).
# Contributing to the project
Pysilfont is developed and maintained by SIL International’s [Writing Systems Technology team ](https://software.sil.org/wstech/), though contributions from anyone are welcome. Pysilfont is copyright (c) 2014-2017 [SIL International](https://www.sil.org) and licensed under the [MIT license](https://en.wikipedia.org/wiki/MIT_License). The project is hosted at [https://github.com/silnrsi/pysilfont](https://github.com/silnrsi/pysilfont).

236
docs/examples.md Normal file
View file

@ -0,0 +1,236 @@
# Pysilfont example scripts
In addition to the main pysilfont [scripts](scripts.md), there are many further scripts under pysilfont/examples and its sub-directories.
They are not maintained in the same way as the main scripts, and come in many categories including:
- Scripts under development
- Examples of how to do things
- Deprecated scripts
- Left-overs from previous development plans!
Note - all FontForge-based scripts need updating, since FontForge (as "FF") is no longer a supported tool for execute()
Some are documented below.
## Table of scripts
| Command | Status | Description |
| ------- | ------ | ----------- |
| [accesslibplist.py](#accesslibplist) | ? | Demo script for accessing fields in lib.plist |
| [chaindemo.py](#chaindemo) | ? | Demo of how to chain calls to multiple scripts together |
| [ffchangeglyphnames](#ffchangeglyphnames) | ? | Update glyph names in a ttf font based on csv file |
| [ffcopyglyphs](#ffcopyglyphs) | ? | Copy glyphs from one font to another, without using ffbuilder |
| [ffremovealloverlaps](#ffremovealloverlaps) | ? | Remove overlap on all glyphs in a ttf font |
| [FFmapGdlNames.py](#ffmapgdlnames) | ? | Write mapping of graphite names to new graphite names |
| [FFmapGdlNames2.py](#ffmapgdlnames2) | ? | Write mapping of graphite names to new graphite names |
| [FLWriteXml.py](#flwritexml) | ? | Outputs attachment point information and notes as XML file for TTFBuilder |
| [FTaddEmptyOT.py](#ftaddemptyot) | ? | Add empty Opentype tables to ttf font |
| [FTMLnorm.py](#ftmlnorm) | ? | Normalize an FTML file |
| [psfaddGlyphDemo.py](#psfaddglyphdemo) | ? | Demo script to add a glyph to a UFO font |
| [psfexpandstroke.py](#psfexpandstroke) | ? | Expands an unclosed UFO stroke font into monoline forms with a fixed width |
| [psfexportnamesunicodesfp.py](#psfexportnamesunicodesfp) | ? | Outputs an unsorted csv file containing the names of all the glyphs in the default layer |
| [psfgenftml.py](#psfgenftml) | ? | generate ftml tests from glyph_data.csv and UFO |
| [psftoneletters.py](#psftoneletters) | ? | Creates Latin script tone letters (pitch contours) |
| [xmlDemo.py](#xmldemo) | ? | Demo script for use of ETWriter |
---
#### accesslibplist
Usage: **` python accesslibplist.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Demo script for accessing fields in lib.plist
---
#### chaindemo
Usage: **` python chaindemo.py ...`**
_([Standard options](docs.md#standard-command-line-options) also apply)_
Demo of how to chain calls to multiple scripts together.
Running
`python chaindemo.py infont outfont --featfile feat.csv --uidsfile uids.csv`
will run execute() against psfnormalize, psfsetassocfeat and psfsetassocuids passing the font, parameters
and logger objects from one call to the next. So:
- the font is only opened once and written once
- there is a single log file produced
---
#### ffchangeglyphnames
Usage: **`ffchangeglyphnames [-i INPUT] [--reverse] ifont [ofont]`**
_([Standard options](docs.md#standard-command-line-options) also apply)_
Update the glyph names in a ttf font based on csv file.
Example usage:
```
ffchangeglyphnames -i glyphmap.csv font.ttf
```
will update the glyph names in the font based on mapping file glyphmap.csv
If \-\-reverse is used, it change names in reverse.
---
#### ffcopyglyphs
Usage: **`ffcopyglyphs -i INPUT [-r RANGE] [--rangefile RANGEFILE] [-n NAME] [--namefile NAMEFILE] [-a] [-f] [-s SCALE] ifont [ofont]`**
_([Standard options](docs.md#standard-command-line-options) also apply)_
_This section is Work In Progress!_
optional arguments:
```
-h, --help show this help message and exit
-i INPUT, --input INPUT
Font to get glyphs from
-r RANGE, --range RANGE
StartUnicode..EndUnicode no spaces, e.g. 20..7E
--rangefile RANGEFILE
File with USVs e.g. 20 or a range e.g. 20..7E or both
-n NAME, --name NAME Include glyph named name
--namefile NAMEFILE File with glyph names
-a, --anchors Copy across anchor points
-f, --force Overwrite existing glyphs in the font
-s SCALE, --scale SCALE
Scale glyphs by this factor
```
---
#### ffremovealloverlaps
Usage: **`ffremovealloverlaps ifont [ofont]`**
_([Standard options](docs.md#standard-command-line-options) also apply)_
Remove overlap on all glyphs in a ttf font
---
#### FFmapGdlNames
Usage: **` python FFmapGdlNames2.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Write mapping of graphite names to new graphite names based on:
- two ttf files
- the gdl files produced by makeGdl run against those fonts
This could be different versions of makeGdl
- a csv mapping glyph names used in original ttf to those in the new font
---
#### FFmapGdlNames2
Usage: **` python FFmapGdlNames.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Write mapping of graphite names to new graphite names based on:
- an original ttf font
- the gdl file produced by makeGdl when original font was produced
- a csv mapping glyph names used in original ttf to those in the new font
- pysilfont's gdl library - so assumes pysilfonts makeGdl will be used with new font
---
#### FLWriteXml
Usage: **` python FLWriteXml.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Outputs attachment point information and notes as XML file for TTFBuilder
---
#### FTaddEmptyOT
Usage: **` python FTaddEmptyOT.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Add empty Opentype tables to ttf font
---
#### FTMLnorm
Usage: **` python FTMLnorm.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Normalize an FTML file
---
#### psfaddGlyphDemo
Usage: **` python psfaddGlyphDemo.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Demo script to add a glyph to a UFO font
---
#### psfexpandstroke
Usage: **`psfexpandstroke infont outfont expansion`**
_([Standard options](docs.md#standard-command-line-options) also apply)_
Expands the outlines (typically unclosed) in an UFO stroke font into monoline forms with a fixed width.
Example that expands the stokes in a UFO font `SevdaStrokeMaster-Regular.ufo` by 13 units on both sides, giving them a total width of 26 units, and writes the result to `Sevda-Regular.ufo`.
```
psfexpandstroke SevdaStrokeMaster-Regular.ufo Sevda-Regular.ufo 13
```
Note that this only expands the outlines - it does not remove any resulting overlap.
---
#### psfexportnamesunicodesfp
Usage: **` python psfexportnamesunicodesfp.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Outputs an unsorted csv file containing the names of all the glyphs in the default layer and their primary unicode values.
Format name,usv
---
#### psfgenftml
Usage: **` python psfgenftml.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
generate ftml tests from glyph_data.csv and UFO
---
#### psftoneletters
Usage: **`psftoneletters infont outfont`**
_([Standard options](docs.md#standard-command-line-options) also apply)_
This uses the parameters from the UFO lib.plist org.sil.lcg.toneLetters key to create Latin script tone letters (pitch contours).
Example usage:
```
psftoneletters Andika-Regular.ufo Andika-Regular.ufo
```
---
#### xmlDemo
Usage: **` python xmlDemo.py ...`**
_([Standard options](docs.md#standard-command-line-options) may also apply)_
Demo script for use of ETWriter

150
docs/fea2_proposal.md Normal file
View file

@ -0,0 +1,150 @@
# Proposed Extensions to FEA
This document describes a macro extension to FEA that will enable it to grow
and support more powerful OpenType descriptions. The proposal is presented as
various syntax extensions to the core FEA syntax.
## Functions
Currently FEA makes no use of parentheses. This may be a conscious decision to
reserve these for later ues. Such parentheses lend themselves perfectly to the
addition of macro functions to the FEA syntax:
```
function = funcname '(' (parameter (',' parameter)*)? ')'
funcname = /[A-Za-z_.][A-Za-z_0-9.]*/
parameter = glyph | glyphlist | classref | value_record | function
| ('"' string '"') | ("{" tokens "}")
tokens = noncurlytoken* | ("{" tokens "}")
glyphlist = '[' glyph* ']'
classref = '@' classname
value_record = number | '<' chars '>'
```
A function call consists of a function name and a parenthesised parameter list,
which may be empty.
and an optional following token list enclosed in braces. The
token list is just that, an unparsed sequence of lexical tokens. The result of
the function is also an unparsed sequence of lexical tokens that are then parsed
and processed as if the function were replaced by a textual representation of
the tokens.
The parameters are parsed, so for example a classref would expand to its
resulting list of glyphs. Likewise a function call result would be parsed to its
single semantic item, it is not parsed as a token list. A value_record is the
widest interpretation of a value record, including an anchor. Basically it is
either a number or anything between < and >.
A function statement is the use of a function result as a statement in the FEA
syntax.
The FEA syntax defines nothing more that functions exist and how they may be
referenced. It is up to a particular FEA processor to supply the functions and
to execute them to resolve them to a token list. It is also up to the particular
FEA processor to report an error or otherwise handle an unknown function
reference. As such this is similar to other programming languages where the
language itself says nothing about what functions exist or what they do. That is
for libraries.
There is one exception. The `include` statement in the core FEA syntax follows
the same syntax, apart from the missing quotation marks around the filename. As
such `include` is not available for use as a function name.
### Sample Implementation
In this section we give a sample implementation based on the FEA library in
fonttools.
Functions are kept in module style namespaces, much like a simplified python module
system. A function name then typically consists of a `modulename.funcname` The
top level module is reserved for the fea processor itself. The following
functions are defined in the top level module (i.e. no modulename.)
#### load
The `load` function takes a path to a file containing python definitions.
Whether this python code is preprocessed for security purposes or not is an open
question. It also takes a modulename as its second parameter.
```
load("path/to/pythonfile.py", "mymodule")
```
The function returns an empty token string but has the effect of loading all the
functions defined in the python file as those functions prefixed by the
modulename, as described above.
#### set
This sets a variable to a token list. Variables are described in a later syntax
extension. The first parameter is the name of a variable. The token list is then
used for the variable expansion.
```
set("distance") { 30 };
```
Other non top level module may be supplied with the core FEA processing module.
#### core.refilter
This function is passed a glyphlist (or via a classref) and a regular
expression. The result is a glyphlist consisting of all the glyphs whose name
matches the regular expression. For example:
```
@csc = core.refilter("\.sc$", @allglyphs)
```
#### core.pairup
This function is passed two classnames, a regular expression and a glyph list.
The result is two class definitions for the two classnames. One class is
of all the glyphs which match the regular expression. The other class is a
corresponding list of glyphs whose name is the same as the matching regular
expression with the matching regular expression text removed. If no such glyph
exists in the font, then neither the name or the glyph matching the regular
expression is included. The resulting classes may therefore be used in a simple
substitution. For example:
```
core.pairup("cnosc", "csc", "\.sc$", [a.sc b.sc fred.sc]);
lookup smallcap {
sub @cnosc by @csc;
} smallcap;
```
Assuming `fred.sc` exists but `fred` does not, this is equivalent to:
```
@cnosc = [a b];
@csc = [a.sc b.sc];
lookup smallcap {
sub @cnosc by @csc;
} smallcap;
```
## Variables
A further extension to the FEA syntax is to add a simple variable expansion. A
variable expands to a token list. Since variables may occur anywhere they need a
syntactic identifier. The proposed identifier is an initial `$`.
```
variable = '$' funcname
```
Variables are expanded at the point of expansion. Since expansion is recursive,
the variable may contain a function call which expands when the variable
expands.
There is no syntax for defining a variable. This is unnatural and may be
revisited if a suitable syntax can be found. Definition is therefore a processor
specific activity.
It is undecided whether undefined variables expand to an empty token list or an
error.

559
docs/feaextensions.md Normal file
View file

@ -0,0 +1,559 @@
# FEA Extensions Current
This document describes the functionality of `psfmakefea` and lists the extensions to fea that are currently supported.
<!-- TOC -->
- [Generated Classes](#generated-classes)
- [Variant glyph classes](#variant-glyph-classes)
- [Ligatures](#ligatures)
- [Statements](#statements)
- [baseclass](#baseclass)
- [Cursive Attachment](#cursive-attachment)
- [Mark Attachment](#mark-attachment)
- [Ligature Attachment](#ligature-attachment)
- [ifinfo](#ifinfo)
- [ifclass](#ifclass)
- [do](#do)
- [SubStatements](#substatements)
- [for](#for)
- [let](#let)
- [forlet](#forlet)
- [if](#if)
- [Examples](#examples)
- [Simple calculation](#simple-calculation)
- [More complex calculation](#more-complex-calculation)
- [Right Guard](#right-guard)
- [Left Guard](#left-guard)
- [Left Kern](#left-kern)
- [Myanmar Great Ya](#myanmar-great-ya)
- [Advance for Ldot on U](#advance-for-ldot-on-u)
- [def](#def)
- [python support](#python-support)
- [kernpairs](#kernpairs)
- [Capabilities](#capabilities)
- [Permit classes on both sides of GSUB type 2 (multiple) and type 4 (ligature) lookups](#permit-classes-on-both-sides-of-gsub-type-2-multiple-and-type-4-ligature-lookups)
- [Processing](#processing)
- [Example](#example)
- [Support classes in alternate lookups](#support-classes-in-alternate-lookups)
- [groups.plist](#groupsplist)
<!-- /TOC -->
## Generated Classes
`psfmakefea` simplifies the hand creation of fea code by analysing the glyphs in the input font, particularly with regard to their names. Names are assumed to conform to the Adobe Glyph List conventions regarding `_` for ligatures and `.` for glyph variants.
### Variant glyph classes
If a font contains a glyph with a final variant (there may be more than one listed for a glyph, in sequence) and also a glyph without that final variant, then `psfmakefea` will create two classes based on the variant name: @c\__variant_ contains the glyph with the variant and @cno\__variant_ contains the glyph without the variant. The two lists are aligned such that a simple classes based replacement will change all the glyphs without the variant into ones with the variant.
For example, U+025B is an open e that occurs in some African languages. Consider a font that contains the glyphs `uni025B` and `uni025B.smcp` for a small caps version of the glyph. `psfmakefea` will create two classes:
```
@c_smcp = [uni025B.scmp];
@cno_smcp = [uni025B];
```
In addition, if this font contains two other glyphs `uni025B.alt`, an alternative shape to `uni025B` and `uni025B.alt.smcp`, the small caps version of the alternate. `psfmakefea` will create the following classes:
```
@c_smcp = [uni025B.scmp uni025B.alt.smcp];
@cno_smcp = [uni025B uni025B.alt];
@c_alt = [uni025B.alt];
@cno_alt = [uni025B];
```
Notice that the classes with multiple glyphs, while keeping the alignment, do not guarantee any particular order of the glyphs in one of the classes. Only that the other class will align its glyph order correctly. Notice also that `uni025B.alt.smcp` does not appear in the `@c_alt` class. This latter behaviour may change.
### Ligatures
Unless instructed on the command line via the `-L` or `--ligmode` option, `psfmakefea` does nothing special with ligatures and treats them simply as glyphs that may take variants. There are four ligature modes. The most commonly used is `-L last`. This says to create classes based on the last components in all ligatures. Thus if the font from the previous section also included `uni025B_acutecomb` and the corresponding small caps `uni025B_acutecomb.smcp`. We also need an `acutecomb`. If the command line included `-L last`, the generated classes would be:
```
@c_smcp = [uni025B.scmp uni025B.alt.smcp uni025B_acutecomb.smcp];
@cno_smcp = [uni025B uni025B.alt uni025B_acutecomb];
@c_alt = [uni025B.alt];
@cno_alt = [uni025B];
@clig_acutecomb = [uni025B_acutecomb];
@cligno_acutecomb = [uni025B];
```
And if the command line option were `-L first`, the last two lines of the above code fragment would become:
```
@clig_uni025B = [uni025B_acutecomb];
@cligno_uni025B = [acutecomb];
```
while the variant classes would remain the same.
There are two other ligaturemodes: `lastcomp` and `firstcomp`. These act like `last` and `first`, but in addition they say that any final variants must be handled differently. Instead of seeing the final variants (those on the last ligature component) as applying to the whole ligature, they are only to be treated as applying to the last component. To demonstrate this we need to add the nonsensical `acutecomb.smcp`. With either `-L last` or `-L first` we get the same ligature classes as above. (Although we would add `acutecomb.smcp` to the `@c_smcp` and `acutecomb` to `@cno_smcp`) With `-L firstcomp` we get:
```
@c_smcp = [uni025B.scmp uni025B.alt.smcp acutecomb.smcp];
@cno_smcp = [uni025B uni025B.alt acutecomb];
@c_alt = [uni025B.alt];
@cno_alt = [uni025B];
@clig_uni025B = [uni025B_acutecomb uni025B_acutecomb.smcp];
@cligno_uni025B = [acutecomb acutecomb.smcp];
```
Notice the removal of `uni025B_acutecomb.smcp` from `@c_smcp`, since `uni025B_acutecomb.smcp` is considered by `-L firstcomp` to be a ligature of `uni025B` and `acutecomb.smcp` there is no overall ligature `uni025B_acutecomb` with a variant `.smcp` that would fit into `@c_smcp`. If we use `-L lastcomp` we change the last two classes to:
```
@clig_acutecomb = [uni025B_acutecomb];
@cligno_acutecomb = [uni025B];
@clig_acutecomb_smcp = [uni025B_acutecomb.smcp];
@cligno_acutecomb_smcp = [un025B];
```
With any `.` in the variant being changed to `_` in the class name.
In our example, if the author wanted to use `-L lastcomp` or `-L firstcomp`, they might find it more helpful to rename `uni025B_acutecomb.smcp` to `uni025B.smcp_acutecomb` and remove the nonsensical `acutecomb.smcp`. This would give, for `-L lastcomp`:
```
@c_smcp = [uni025B.scmp uni025B.alt.smcp];
@cno_smcp = [uni025B uni025B.alt];
@c_alt = [uni025B.alt];
@cno_alt = [uni025B];
@clig_acutecomb = [uni025B_acutecomb uni025B.smcp_acutecomb];
@cligno_acutecomb = [uni025B uni025B.smcp];
```
and for `-L firstcomp`, the last two classes become:
```
@clig_uni025B = [uni025B_acutecomb];
@cligno_uni025B = [acutecomb];
@clig_uni025B_smcp = [uni025B.smcp_acutecomb];
@cligno_uni025B_smcp = [acutecomb];
```
## Statements
### baseclass
A baseclass is the base equivalent of a markclass. It specifies the position of a particular class of anchor points on a base, be that a true base or a mark base. The syntax is the same as for a markclass, but it is used differently in a pos rule:
```
markClass [acute] <anchor 350 0> @TOP_MARKS;
baseClass [a] <anchor 500 500> @BASE_TOPS;
baseClass b <anchor 500 750> @BASE_TOPS;
feature test {
pos base @BASE_TOPS mark @TOP_MARKS;
} test;
```
Which is the functional equivalent of:
```
markClass [acute] <anchor 350 0> @TOP_MARKS;
feature test {
pos base [a] <anchor 500 500> mark @TOP_MARKS;
pos base b <anchor 500 750> mark @TOP_MARKS;
} test;
```
It should be borne in mind that both markClasses and baseClasses can also be used as normal glyph classes and as such use the same namespace.
The baseClass statement is a high priority need in order to facilitate auto generation of attachment point information without having to create what might be redundant lookups in the wrong order.
Given a set of base glyphs with attachment point A and marks with attachment point \_A, psfmakefea will generate the following:
- baseClass A - containing all bases with attachment point A
- markClass \_A - containing all marks with attachment point \_A
- baseClass A\_MarkBase - containing all marks with attachment point A
#### Cursive Attachment
Cursive attachment involves two base anchors, one for the entry and one for the exit. We can extend the use of baseClasses to support this, by passing two baseClasses to the pos cursive statement:
```
baseClass meem.medial <anchor 700 50> @ENTRIES;
baseClass meem.medial <anchor 0 10> @EXITS;
feature test {
pos cursive @ENTRIES @EXITS;
} test;
```
Here we have two base classes for the two anchor points, and the pos cursive processing code works out which glyphs are in both classes, and which are in one or the other and generates the necessary pos cursive statement for each glyph. I.e. there will be statements for the union of the two classes but with null anchors for those only in one (according to which baseClass they are in). This has the added advantage that any code generating baseClasses does not need to know whether a particular attachment point is being used in a cursive attachment. That is entirely up to the user of the baseClass.
#### Mark Attachment
The current mark attachment syntax is related to the base mark attachment in that the base mark has to be specified explicitly and we cannot currently use a markclass as the base mark in a mark attachment lookup. We can extend the mark attachment in the same way as we extend the base attachment, by allowing the mark base to be a markclass. Thus:
```
pos mark @MARK_BASE_CLASS mark @MARK_MARK_CLASS;
```
Would expand out to a list of mark mark attachment rules.
#### Ligature Attachment
Ligature attachment involves all the attachments to a ligature in a single rule. Given a list of possible ligature glyphs, the ligature positioning rule has been extended to allow the use of baseClasses instead of the base anchor on the ligature. For a noddy example:
```
baseClass a <anchor 200 200> @TOP_1;
baseClass fi <anchor 200 0> @BOTTOM_1;
baseClass fi <anchor 400 0> @BOTTOM_2;
markClass acute <anchor 0 200> @TOP;
markClass circumflex <anchor 200 0> @BOTTOM;
pos ligature [a fi] @BOTTOM_1 mark @BOTTOM @TOP_1 mark @TOP
ligComponent @BOTTOM_2 mark @BOTTOM;
```
becomes
```
pos ligature a <anchor 200 200> mark @TOP
ligComponent <anchor NULL>;
pos ligature fi <anchor 200 0> mark @BOTTOM
ligComponent <anchor 400 0> mark @BOTTOM;
```
### ifinfo
This statement initiates a block either of statements or within another block. The block is only processed if the ifinfo condition is met. ifinfo takes two parameters. The first is a name that is an entry in a fontinfo.plist. The second is a string containing a regular expression that is matched against the given value in the fontinfo.plist. If there is a match, the condition is considered to be met.
```
ifinfo(familyName, "Doulos") {
# statements
}
```
Notice the lack of a `;` after the block close.
ifinfo acts as a kind of macro, this means that the test is executed in the parser rather than collecting everything inside the block and processing it later like say the `do` statement. Notice that if you want to do something more complex than a regular expression test, then you may need to use a `do` statement and the `info()` function.
### ifclass
This statement initiates a block either of statements or within another block. The block is only processed if the given @class is defined and contains at least one glyph.
```
ifclass(@oddities) {
# statements
}
```
Notice the lack of a `;` after the block close.
### do
The `do` statement is a means of setting variables and repeating statement groups with variable expansion. A `do` statement is followed by various substatements that are in effect nested statements. The basic structure of the `do` statement is:
`do` _substatement_ _substatement_ _..._ [ `{` _statements_ `}` ]
Where _statements_ is a sequence of FEA statements. Within these statements, variables may be referenced by preceding them with a `$`. Anything, including statement words, can be the result of variable expantion. The only constraints are:
- The item expands to one or more complete tokens. It cannot be joined to something preceding or following it to create a single name, token, whatever.
In effect a `{}` type block following a `for` or `let` substatement is the equivalent of inserting the substatement `if True;` before the block.
#### SubStatements
Each substatement is terminated by a `;`. The various substatements are:
##### for
The `for` substatement is structured as:
`for` _var_ `=` _glyphlist_ `;`
This creates a variable _var_ that will iterate over the _glyphlist_.
With the addition of `forlet` (see below), there is also `forgroup` that is a synonym for the `for` substatement defined here.
##### let
The `let` substatement executes a short python expression (via `eval`), storing the result in the given variable, or variable list. The structure of the substatement is:
`let` _var_ [`,` _var_]* `=` _expression_ `;`
There are various python functions that are especially supported, along with the builtins. These are:
| Function | Parameters | Description |
|-------------|-------------|------------------------|
| ADVx | _glyphname_ | Returns the advanced width of the given glyph |
| allglyphs | | Returns a list of all the glyph names in the font |
| APx | _glyphname_, "_apname_" | Returns the x coordinate of the given attachment point on the given glyph |
| APy | _glyphname_, "_apname_" | Returns the y coordinate of the given attachment point on the given glyph |
| feaclass | _classname_ | Returns a list of the glyph names in a class as a python list |
| info | _finfoelement_ | Looks up the entry in the fontinfo plist and returns its value |
| kerninfo | | Returns a list of tuples (left, right, kern_value) |
| opt | _defined_ | Looks up a given -D/--define variable. Returns empty string if missing |
| MINx | _glyphname_ | Returns the minimum x value of the bounding box of the glyph |
| MINy | _glyphname_ | Returns the minimum y value of the bounding box of the glyph |
| MAXx | _glyphname_ | Returns the maximum x value of the bounding box of the glyph |
| MAXy | _glyphname_ | Returns the maximum y value of the bounding box of the glyph |
See the section on python in the `def` command section following.
##### forlet
The `for` substatement only allows iteration over a group of glyphs. There are situations in which someone would want to iterate over a true python expression, for example, over the return value of a function. The `forlet` substatement is structured identically to a `let` substatement, but instead of setting the variable once, the following substatements are executed once for each value of the expression, with the variable set to each in turn. For example:
```
def optlist(*alist) {
if len(alist) > 0:
for r in optlist(*alist[1:]):
yield [alist[0]] + r
yield r
else:
yield alist
} optlist;
lookup example {
do
forlet l = optlist("uni17CC", "@coeng_no_ro", "[uni17C9 uni17CA]", "@below_vowels", "@above_vowels");
let s = " ".join(l)
{
sub uni17C1 @coeng_ro @base_cons $s uni17B8' lookup insert_dotted_circle;
sub uni17C1 @base_cons $s uni17B8' lookup insert_dotted_circle;
}
} example;
```
This examples uses a `def` statement as defined below. The example produces rules for each of the possible subsequences of the optlist parameters, where each element is treated as being optional. It is a way of writing:
```
sub uni17C1 @base_cons uni17CC? @coeng_no_ro [uni17C9 uni17CA]? @below_vowels? @above_vowels? uni17B8' lookup insert_dotted_circle;
```
The structure of a `forlet` substatement is:
`forlet` _var_ [`,` _var_]* `=` _expression_ `;`
A `forlet` substatement has the same access to functions that the `let` statement has, included those listed above under `let`.
##### if
The `if` substatement consists of an expression and a block of statements. `if` substatements only make sense at the end of a sequence of substatements and are executed at the end of the `do` statement, in the order they occur but after all other `for` and `let` substatements. The expression is calculated and if the result is True then the _statements_ are expanded using variable expansion.
`if` _expression_ `;` `{` _statements_ `}`
There can be multiple `if` substatements, each with their own block, in a `do` statement.
#### Examples
The `do` statement is best understood through some examples.
##### Simple calculation
This calculates a simple offset shift and creates a lookup to apply it:
```
do let a = -int(ADVx("u16F61") / 2);
{
lookup left_shift_vowel {
pos @_H <$a 0 0 0>;
} left_shift_vowel;
}
```
Notice the lack of iteration here.
##### More complex calculation
This calculates the guard spaces on either side of a base glyph in response to applied diacritics.
```
lookup advance_base {
do for g = @H;
let a = APx(g, "H") - ADVx(g) + int(1.5 * ADVx("u16F61"));
let b = int(1.5 * ADVx("u16F61")) - APx(g, "H");
let c = a + b;
{
pos $g <$b 0 $c 0>;
}
} advance_base;
```
##### Right Guard
It is often desirable to give a base character extra advance width to account for a diacritic hanging over the right hand side of the glyph. Calculating this can be very difficult by hand. This code achieves this:
```
do for b = @bases;
for d = @diacritics;
let v = (ADVx(d) - APx(d, "_U")) - (ADVx(b) - APx(b, "U"));
if v > 0; {
pos $b' $v $d;
}
```
##### Left Guard
A corresponding guarding of space for diacritics may be done on the left side of a glyph:
```
do for b = @bases;
for d = @diacritics;
let v = APx(d, "_U") - APx(b, "U");
if v > 0; {
pos $b' <$v 0 $v 0> $d;
}
```
##### Left Kern
Consider the case where someone has used an attachment point as a kerning point. In some context they want to adjust the advance of the left glyph based on the position of the attachment point in the right glyph:
```
do for r = @rights;
let v = APx(r, "K"); {
pos @lefts' $v $r;
pos @lefts' $v @diacritics $r;
}
```
##### Myanmar Great Ya
One obscure situation is the Great Ya (U+103C) in the Myanmar script, that visual wraps around the following base glyph. The great ya is given a small advance to then position the following consonant glyph within it. The advance of this consonant needs to be enough to place the next character outside the great ya. So we create an A attachment point on the great ya to emulate this intended final advance. Note that there are many variants of the great ya glyph. Thus:
```
do for y = @c103C_nar;
for c = @cCons_nar;
let v = APx(y, "A") - (ADVx(y) + ADVx(c));
if v > 0; {
pos $y' $v $c;
}
do for y = @c103C_wide;
for c = @cCons_wide;
let v = APx(y, "A") - (ADVx(y) + ADVx(c));
if v > 0; {
pos $y' $v $c;
}
```
##### Advance for Ldot on U
This example mirrors that used in the proposed [`setadvance`](feax_future.md#setadvance) statement. Here we want to add sufficient advance on the base to correspond to attaching an u vowel which in turn has a lower dot attached to it.
```
do for b = @cBases;
for u = @cLVowels;
let v = APx(b, "L") - APx(u, "_L") + APx(u, "LD") - APx("ldot", "_LD") + ADVx("ldot") - ADVx(b);
if v > 0; {
pos $b' $v $u ldot;
}
```
### def
The `def` statement allows for the creation of python functions for use in `let` substatements of the `do` statement. The syntax of the `def` statement is:
```
def <fn>(<param_list>) {
... python code ...
} <fn>;
```
The `fn` must conform to a FEA name (not starting with a digit, etc.) and is repeated at the end of the block to mark the end of the function. The parameter is a standard python parameter list and the python code is standard python code, indented as if under a `def` statement.
#### python support
Here and in `let` and `forlet` substatements, the python that is allowed to be executed is limited. Only a subset of functions from builtins is supported and the `__` may not occur in any attribute. This is to stop people escaping the sandbox in which python code is interpreted. The `math` and `re` modules are also included along with the functions available to a `let` and `forlet` substatement. The full list of builtins supported are:
```
True, False, None, int, float, str, abs, bool, dict, enumerate, filter, hex, isinstance, len, list,
map, max, min, ord, range, set, sorted, sum, tuple, type, zip
```
### kernpairs
The `kernpairs` statement expands all the kerning pairs in the font into `pos` statements. For example:
```
lookup kernpairs {
lookupflag IgnoreMarks;
kernpairs;
} kernpairs;
```
Might produce:
```
lookup kernpairs {
lookupflag IgnoreMarks;
pos @MMK_L_afii57929 -164 @MMK_R_uniA4F8;
pos @MMK_L_uniA4D1 -164 @MMK_R_uniA4F8;
pos @MMK_L_uniA4D5 -164 @MMK_R_afii57929;
pos @MMK_L_uniA4FA -148 @MMK_R_space;
} kernpairs;
```
Currently, kerning information is only available from .ufo files.
## Capabilities
### Permit classes on both sides of GSUB type 2 (multiple) and type 4 (ligature) lookups
Adobe doesn't permit compact notation using groups in 1-to-many (decomposition) rules e.g:
```
sub @AlefPlusMark by absAlef @AlefMark ;
```
or many-to-1 (ligature) rules, e.g.:
```
sub @ShaddaKasraMarks absShadda by @ShaddaKasraLigatures ;
```
This is implemented in FEAX as follows.
#### Processing
Of the four simple (i.e., non-contextual) substitution lookups, Types 2 and 4
are the only ones using the 'by' keyword that have a *sequence* of glyphs or
classes on one side of the rule. The other side will, necessarily, contain a
single term -- which Adobe currently requires to be a glyph. For convenience of
expression, we'll call the sides of the rule the *sequence side* and the *singleton side*.
* Non-contextual substitution
* Uses the 'by' keyword
* Singleton side references a glyph class.
Such rules are expanded by enumerating the singleton side class and the corresponding
class(es) on the sequence side and writing a set of Adobe-compliant rules to give
the same result. It is an error if the singleton and corresponding classes do
not have the same number of glyphs.
#### Example
Given:
```
@class1 = [ g1 g2 ] ;
@class2 = [ g1a g2a ] ;
```
then
```
sub @class1 gOther by @class2 ;
```
would be rewritten as:
```
sub g1 gOther by g1a ;
sub g2 gOther by g2a ;
```
### Support classes in alternate lookups
The default behaviour in FEA is for a `sub x from [x.a x.b];` to only allow a single glyph before the `from` keyword. But it is often useful to do things like: `sub @a from [@a.lower @a.upper];`. Feax supports this by treating the right hand side list of glyphs as a single list and dividing it equally by the list on the left. Thus if `@a` is of length 3 then the first 3 glyphs in the right hand list will go one each as the first alternate for each glyph in `@a`, then the next 3 go as the second alternate, and so on until they are all consumed. If any are left over in that one of the glyphs ends up with a different number of alternates to another, then an error is given.
### groups.plist
If a .ufo file contains a `groups.plist` file, the groups declared there are propagated straight through to the output file and can be referenced within a source file.

283
docs/feax_future.md Normal file
View file

@ -0,0 +1,283 @@
# FEA Extensions Future
## Introduction
This document is where people can dream of the extensions they would like to see
added to FEA. Notice that any extensions need to be convertible back to normal FEA
so shouldn't do things that can't be expressed in FEA.
As things get implemented from here, they will be moved to feaextensions.md. There
are no guaranteees that what is in here, will end up in psfmakefea.
The various features listed here are given priorities:
| Level | Priority
|------|-------------
| 1 | Intended to be implemented
| 2 | Probably will be implemented but after priority 1 stuff
| 3 | Almost certainly won't be implemented
There are a number of possible things that can be added to FEA, the question is whether they are worth adding in terms of meeting actual need (remove from this list if added to the rest of the document):
* classsubtract() classand() functions
* classand(x, y) = classsubtract(x, (classsubtract(x, y))
* classbuild(class, "$.ext") builds one class out of another. What if something is missing? Or do we just build those classes on the fly from make_fea and glyph name parsing?
## Statements
Statements are used to make rules, lookups, etc.
### setadvance
Priority: 3 (since the do statement has a higher priority and covers this)
This function does the calculations necessary to adjust the advance of a glyph based on information of attachment points, etc. The result is a single shift on each of the glyphs in the class. The syntax is:
```
setadvance(@glyphs, APName [, attachedGlyph[, APName, attachedGlyph [...]]])
```
In effect there are two modes for this function. The first only has two parameters
and shifts the advance from its default designed position to the x coordinate of
the given attachment point. The second mode adds extra glyphs. The advance is moved
to the advance of the attachedGlyph assuming the base has the other glyphs chained
attached at their given APs. An AP may be a number in which case that is the
x coordinate of the AP that will be used.
Typically there will be only one of these per lookup, unless the classes referenced
are non overlapping.
The statement only triggers if the resulting advance is greater than the current
advance. Thus some glyphs may not have a statement created for them. I.e. all
values in the lookup will be positive.
#### Examples
These examples also act as motivating use cases.
##### Nokyung
In Nokyung there is a need to kern characters that do not descend below the baseline closer to glyphs with a right underhang. This can be done through kerning pairs or we could add an attachment point to the glyphs with the right underhang and contextual adjust their advances to that position. The approach of using an AP to do kerning is certainly quirky and few designers would go that route. The contextual lookup would call a lookup that just does the single adjustment. Consider the AP to be called K (for kern). The fea might look like:
```
lookup overhangKernShift {
setadvance(@overhangs, K);
} overhangKernShift;
```
And would expand, potentially, into
```
lookup overhangKernShift {
@overhangs <-80>;
} overhangKernShift;
```
Not much, but that is because in Nokyung the overhanging glyphs all have the same overhang. If they didn't, then the list could well expand with different values for each glyph in the overhangs class. In fact, a simple implementation would do such an expansion anyway, while a more sophisticated implementation would group the results into ad hoc glyph lists.
##### Myanmar
An example from Myanmar is where a diacritic is attached such that the diacritic overhangs the right hand side of the base glyph and we want to extend the advance of the base glyph to encompass the diacritic. This is a primary motivating example for this statement. Such a lookup might read:
```
lookup advanceForLDotOnU {
setadvance(@base, L, uvowel, LD, ldot);
} advanceForLDotOnU;
```
Which transforms to:
```
lookup advanceForLDotOnU {
ka <120>;
kha <80>;
# …
} advanceForLDotOnU;
```
##### Miao
Miao is slightly different in that the advance we want to use is a constant,
partly because calculating it involves a sequence of 3 vowel widths and you end up
with a very long list of possible values and lookups for each one:
```
lookup advancesShortShortShort {
setadvance(@base, 1037);
} advancesShortShortShort;
```
#### Issues
* Do we want to use a syntax more akin to that used for composites, since that is, in effect, what we are describing: make the base have the advance of the composite?
* Do we want to change the output to reflect the sequence so that there can be more statements per lookup?
* The problem is that then you may want to skip intervening non-contributing glyphs (like upper diacritics in the above examples), which you would do anyway from the contextual driving lookup, but wouldn't want to have to do in each situation here.
* It's a bit of a pain that in effect there is only one setadvance() per lookup. It would be nice to do more.
* Does this work (and have useful meaning) in RTL?
* Appears to leave the base glyph *position* unchanged. Is there a need to handle, for example in LTR scripts, LSB change for a base due to its diacritics? (Think i-tilde, etc.)
### move
Priority: 2
The move semantic results in a complex of lookups. See this [article](https://github.com/OpenType/opentype-layout/blob/master/docs/ligatures.md) on how to implement a move semantic successfully in OpenType. As such a move semantic can only be expressed as a statement at the highest level since it creates lookups. The move statement takes a number of parameters:
```
move lookup_basename, skipped, matched;
```
The *lookup_basename* is a name (unadorned string) prefix that is used in the naming of the lookups that the move statement creates. It also allows multiple move statements to share the same lookups where appropriate. Such lookups can be referenced by contextual chaining lookups. The lookups generated are:
| | |
| ---------------------------- | -------------------------------------------------- |
| lookup_basename_match | Contextual chaining lookup to drive the sublookups |
| lookup_basename_pres_matched | Converts skipped(1) to matched + skipped(1) |
| lookup_basename_pref_matched | Converts skipped(1) to matched + skipped(1) + matched |
| lookup_basename_back | Converts skipped(-1) + matched to skipped(-1). |
Multiple instances of a move statement that use the same *lookup_basename* will correctly merge the various rules in the the lookups created since often at least parts of the *skipped* or *matched* will be the same across different statements.
Since lookups may be added to, extra contextual rules can be added to the *lookup_basename*_match.
*skipped* contains a sequence of glyphs (of minimum length 1), where each glyph may be a class or whatever. The move statement considers both the first and last glyph of this sequence when it comes to the other lookups it creates. *skipped(1)* is the first glyph in the sequence and *skipped(-1)* is the last.
*matched* is a single glyph that is to be moved. There needs to be a two lookups for each matched glyph.
Notice that only *lookup_basename*_matched should be added to a feature. The rest are sublookups and can be in any order. The *lookup_basename*_matched lookup is created at the point of the first move statement that has a first parameter of *lookup_basename*.
#### Examples
While there are no known use cases for this in our fonts at the moment, this is an important statement in terms of showing how complex concepts of wider interest can be implemented as extensions to fea.
##### Myanmar
Moving prevowels to the front of a syllable from their specified position in the sequence, in a DFLT processor is one such use of a move semantic:
```
move(pv, @cons, my-e);
move(pv, @cons @medial, my-e);
move(pv, @cons @medial @medial, my-e);
move(pv, @cons @medial @medial @medial, my-e);
move(pv, @cons, my-shane);
move(pv, @cons, @medial, my-shane);
```
This becomes:
```
lookup pv_pres_my-e {
sub @cons by my-e @cons;
} pv_pres_my-e;
lookup pv_pref_my-e {
sub @cons by my-e @cons my-e;
} pv_pref_my-e;
lookup pv_back {
sub @cons my-e by @cons;
sub @medial my-e by @medial;
sub @cons my-shane by @cons;
sub @medial my-shane by @medial;
} pv_back;
lookup pv_match {
sub @cons' lookup pv_pres-my-e my-e' lookup pv_back;
sub @cons' lookup pv_pref-my-e @medial my-e' lookup pv_back;
sub @cons' lookup pv_pref-my-e @medial @medial my-e' lookup pv_back;
sub @cons' lookup pv_pref-my-e @medial @medial @medial my-e' lookup pv_back;
sub @cons' lookup pv_pres-my-shane my-shane' lookup pv_back;
sub @cons' lookup pv_pref-my-shane @medial my-shane' lookup pv_back;
} pv_match;
lookup pv_pres_my-shane {
sub @cons by my-shane @cons;
} pv_pres_my-shane;
lookup pv_pref_my-shane {
sub @cons by my-shane @cons my-shane;
} pv_pref_my-shane;
```
##### Khmer Split Vowels
Khmer has a system of split vowels, of which we will consider a very few:
```
lookup presplit {
sub km-oe by km-e km-ii;
sub km-ya by km-e km-yy km-ya.sub;
sub km-oo by km-e km-aa;
} presplit;
move(split, @cons, km-e);
move(split, @cons @medial, km-e);
```
## Functions
Functions may be used in the place of a glyph or glyph class and return a list of glyphs.
### index
Priority: 2
Used in rules where the expansion of a rule results in a particular glyph from a class being used. Where two classes need to be synchronised, and which two classes are involved, this function specifies the rule element that drives the choice of glyph from this class. This function is motivated by the Keyman language. The parameters of index() are:
```
index(slot_index, glyphclass)
```
*slot_index* considers the rule as two sequences of slots, each slot referring to one glyph or glyphclass. The first sequence is on the left hand side of the rule and the second on the right, with the index running sequentially from one sequence to the other. Thus if a rule has 2 slots on the left hand side and 3 on the right, a *slot_index* of 5 refers to the last glyph on the right hand side. *Slot_index* values start from 1 for the first glyph on the left hand side.
What makes an index() function difficult to implement is that it requires knowledge of its context in the statement it occurs in. This is tricky to implement since it is a kind of layer violation. It doesn't matter how an index() type function is represented syntactically, the same problem applies.
### infont
Priority: 2
This function filters the glyph class that is passed to it, and returns only those glyphs, in glyphclass order, which are actually present in the font being compiled for. For example:
```
@cons = infont([ka kha gha nga]);
```
## Capabilities
### Permit multiple classes on RHS of GSUB type 2 (multiple) and the LHS of type 4 (ligature) lookups
Priority: 2
#### Slot correspondence
In Type 2 (multiple) substitutions, the LHS will be the singleton case and the RHS will be the sequence. In normal use-cases exactly one slot in the RHS will be a class -- all the others will be glyphs -- in which case that class and the singleton side class correspond.
If more than one RHS slot is to contain a class, then the only logical meaning is that all such classes must also correspond to the singleton class in the LHS, and will be expanded (along with the singleton side class) in parallel. Thus all the classes must have the same number of elements.
In Type 4 (ligature) substitutions, the RHS will be the singleton class. In the case that the LHS (sequence side) of the rule has class references in more than one slot, we need to identify which slot corresponds to the singleton side class. Some alternatives:
* Pick the slot that, when the classes are flattened, has the same number of glyphs as the class on the singleton side. It is possible that there is more than one such slot, however.
* Add a notation to the rule. Graphite uses the $n modifier on the RHS to identify the corresponding slot (in the context), which we could adapt to FEA as:
```
sub @class1 @class2 @class3 by @class4$2 ;
```
Alternatively, since there can be only one such slot, we could use a simpler notation by putting something like the $ in the LHS:
```
sub @class1 @class2$ @class3 by @class4 ;
```
[This won't look right to GDL programmers, but makes does sense for OT code]
* Extra syntactic elements at the lexical level are hard to introduce. Instead a function such as:
```
sub @class1 @class2 @class3 by index(2, @class4);
```
Would give the necessary interpretation. See the discussion of the index() function for more details.
Note that the other classes in the LHS of ligature rules do not need further processing since FEA allows such classes.
#### Nested classes
We will want to expand nested classes in a way (i.e., depth or breadth first) that is compatible with Adobe. **Concern:** Might this be different than Graphite? Is there any difference if one says always expand left to right? [a b [c [d e] f] g] flattens the same as [[[a b] c d] e f g] or whatever. The FontTools parser does not support nested glyph classes. To what extent are they required?

179
docs/parameters.md Normal file
View file

@ -0,0 +1,179 @@
# Pysilfont parameters
In addition to normal command-line arguments (see [scripts.md](scripts.md) and [Standard Command-line Options](docs.md#standard-command-line-options)), Pysilfont supports many other parameters that can be changed either on the command-line or by settings in a config file. For UFO fonts there is also an option to set parameters within the UFO.
See [List of Parameters](#list-of-parameters) for a full list, which includes the default values for each parameter.
# Setting parameters
Parameters can be set in multiple ways
1. Default values are set by the core.py Pysilfont module - see [List of Parameters](#list-of-parameters)
1. Standard values for a project can be set in a pysilfont.cfg [config file](#config-file)
1. For UFO fonts, font-specific values can be set within the [lib.plist](#lib-plist) file
1. On the command line \- see next section
Values set by later methods override those set by earlier methods.
(Scripts can also change some values, but they would normally be written to avoid overwriting command-line values)
## Command line
For script users, parameters can be set on the command line with -p, for example:
```
psfnormalize test.ufo -p scrlevel=V -p indentIncr=" "
```
would increase the screen reporting level to Verbose and change the xml indent from 2 spaces to 4 spaces.
If a parameter has multiple values, enter them separated with commas but no spaces, eg:
`-p glifElemOrder=unicode,advance,note,image,guideline,anchor,outline,lib`
## Config file
If pysilfont.cfg exists in the same directory as the first file specified on the command line (typically the font being processed) then parameters will be read from there.
The format is a [ConfigParser](https://docs.python.org/2/library/configparser.html) config file, which is similar structure to a Windows .ini file.
Lines starting with # are ignored, as are any blank lines.
Example:
```
# Config file
[logging]
scrlevel: I
[outparams]
indentIncr: ' '
glifElemOrder: unicode,advance,note,image,guideline,anchor,outline,lib
```
The section headers are backups, logging, outparams and ufometadata.
In a font project with multiple UFO fonts in the same folder, all would use a single config file.
## lib plist
If, with a UFO font, org.sil.pysilfontparams exists in lib.plist, parameter values held in an array will be processed, eg
```
<key>org.sil.pysilfontparams</key>
<array>
<indentIncr>\t</indentIncr>
<glifElemOrder>lib,unicode,note,image,guideline,anchor,outline,advance</glifElemOrder>
</array>
```
Currently only font output parameters can be changed via lib.plist
## List of parameters
| Parameter | Default | Description | Notes |
| -------- | -------- | --------------------------------------------- | ------------------------------------- |
| **Reporting** | | | To change within a script use <br>`logger.<parameter> = <value>`|
| scrlevel | P | Reporting level to screen. See [Reporting](docs.md#reporting) for more details | -q, --quiet option sets this to S |
| loglevel | W | Reporting level to log file | |
| **Backup** (font scripts only) | | | |
| backup | True | Backup font to subdirectory | If the original font is being updated, make a backup first |
| backupdir | backups | Sub-directory name for backups | |
| backupkeep | 5 | Number of backups to keep | |
| **Output** (UFO scripts only) | | | To change in a script use <br>`font.outparams[<parameter>] = <value>` |
| indentFirst | 2 spaces | Increment for first level in xml | |
| indentIncr | 2 spaces | Amount to increment xml indents | |
| indentML | False | Indent multi-line text items | (indenting really messes some things up!) |
| plistIndentFirst | Empty string | Different initial indent for plists | (dict is commonly not indented) |
| sortDicts | True | sort all plist dicts | |
| precision | 6 | decimal precision | |
| renameGlifs | True | Name glifs with standard algorithm | |
| UFOversion | (existing) | | Defaults to the version of the UFO when opened |
| format1Glifs | False| Force output of format 1 glifs | Includes UFO2-style anchors; for use with FontForge |
| floatAttribs | (list of attributes in the spec that hold numbers and are handled as float) | Used to know if precision needs setting. | May need items adding for lib data |
| intAttribs | (list of attributes in the spec that hold numbers and handled as integer) | | May need items adding for lib data |
| glifElemOrder | (list of elements in the order defined in spec) | Order for outputting elements in a glif | |
| attribOrders | (list of attribute orders defined in spec) | Order for outputting attributes in an element. One list per element type | When setting this, the parameter name is `attribOrders.<element type>`. Currently only used with attribOrders.glif |
| **ufometadata** (ufo scripts only) | | | |
| checkfix | check | Metadata check & fix action | If set to "fix", some values updated (or deleted). Set to "none" for no metadata checking |
| More may be added... | |
## Within basic scripts
### Accessing values
If you need to access values of parameters or to see what values have been set on the command line you can look at:
- args.paramsobj.sets[“main”]
- This is a dictionary containing the values for **all** parameters listed above. Where they have been specified in a config file, or overwritten on the command line, those values will be used. Otherwise the default values listed above will be used
- args.params
- This is a dictionary containing any parameters specified on the command line with -p.
Within a UFO Ufont object, use font.paramset, since this will include any updates as a result parameter values set in lib.plist.
In addition to the parameters in the table above, two more read-only parameters can be accessed by scripts - “version” and “copyright” - which give the pysilfont library version and copyright info, based on values in core.py headers.
### Updating values
Currently only values under Output can be set via scripts, since Backup and Reporting parameters are processed by execute() prior to the script being called. For example:
```python
font.paramset[“precision”] = 9
```
would set the precision parameter to 9.
Note that, whilst reporting _parameters_ can’t be set in scripts, _reporting levels_ can be updated by setting values in the args.logger() object, eg `args.logger.scrlevel = “W”.`
# Technical
_Note the details below are probably not needed just for developing scripts..._
## Basics
The default for all parameters are set in core.py as part of the parameters() object. Those for **all** pysilfont library modules need to be defined in core.py so that execute() can process command-line arguments without needing information from other modules.
Parameters are passed to scripts via a parameters() object as args.paramsobj. This contains several parameter sets, with “main” being the standard one for scripts to use since that contains the default parameters updated with those (if any) from the config file then the same for any command-line values.
Parameters can be accessed from the parameter set by parameter name, eg paramsobj.sets[“main”][“loglevel”].
Although parameters are split into classes (eg backup, logging), parameter names need to be unique across all groups to allow simple access by name.
If logging set set to I or V, changes to parameter values (eg config file values updating default values) are logged.
There should only be ever a single parameters() object used by a script.
## Paramobj
In addition to the paramsets, the paramobj also contains
- classes:
- A dictionary keyed on class, returning a list of parameter names in that class
- paramclass:
- A dictionary keyed on parameter name, returning the class of that parameter
- lcase:
- A dictionary keyed on lowercase version of parameter name returning the parameter name
- type:
- A dictionary keyed on parameter name, returning the type of that parameter (eg str, boolean, list)
- listtype:
- For list parameters, a dictionary keyed on parameter name, returning the type of that parameters in the list
- logger:
- The logger object for the script
## Parameter sets
These serve two purposes:
1. To allow multiple set of parameter values to be used - eg two different fonts might have different values in the lib.plist
1. To keep track of the original sets of parameters (“default”, “config file” and “command line”) if needed. See UFO specific for an example of this need.
Additional sets can be added with addset() and one set can be updated with values from another using updatewith(), for example, to create the “main” set, the following code is used:
```
params.addset("main",copyset = "default") # Make a copy of the default set
params.sets["main"].updatewith("config file") # Update with config file values
params.sets["main"].updatewith("command line") # Update with command-line values
```
## UFO-specific
The parameter set relevant to a UFO font can be accessed by font.paramset, so font.paramset[“loglevel"] would access the loglevel.
In ufo.py there is code to cope with two complications:
1. If a script is opening multiple fonts, in they could have different lib.plist values so font-specific parameter sets are needed
1. The parameters passed to ufo.py include the “main” set which has already had command-line parameters applied. Any in lib.plist also need to be applied, but can’t simply be applied to “main” since command-line parameters should take precedence over lib.plist ones
To ensure unique names, the parameter sets are created using the full path name of the UFO. Then font.paramset is set to point to this, so scripts do not need to know the underlying set name.
To apply the parameter sets updates in the correct order, ufo.py does:
1. Create a new paramset from any lib parameters present
1. Update this with any command line parameters
1. Create the paramset for the font by copying the “main” paramset
1. Update this with the lib paramset (which has already been updated with command line values in step 2)
## Adding another parameter or class
If there was a need to add another parameter or class, all that should be needed is to add that to defparams in the \_\_init\_\_() of parameters() in core.py. Ensure the new parameter is case-insensitively unique.
If a class was Ufont-specific and needed to be supported within lib.plist, then ufo.py would also need updating to handle that similarly to how it now handles outparams and ufometadata.

1565
docs/scripts.md Normal file
View file

File diff suppressed because it is too large Load diff

342
docs/technical.md Normal file
View file

@ -0,0 +1,342 @@
# Pysilfont Technical Documentation
This section is for script writers and developers.
See [docs.md](docs.md) for the main Pysilfont user documentation.
# Writing scripts
The Pysilfont modules are designed so that all scripts operate using a standard framework based on the execute() command in core.py. The purpose of the framework is to:
- Simplify the writing of scripts, with much work (eg parameter parsing, opening fonts) being handled there rather than within the script.
- Provide a consistent user interface for all Pysilfont command-line scripts
The framework covers:
- Parsing arguments (parameters and options)
- Defaults for arguments
- Extended parameter support by command-line or config file
- Producing help text
- Opening fonts and other files
- Outputting fonts (including normalization for UFO fonts)
- Initial error handling
- Reporting (logging) - both to screen and log file
## Basic use of the framework
The structure of a command-line script should be:
```
<header lines>
<general imports, if any>
from silfont.core import execute
argspec = [ <parameter/option definitions> ]
def doit(args):
<main script code>
return <output font, if any>
<other function definitions>
def cmd() : execute(Tool,doit, argspec)
if __name__ == "__main__": cmd()
```
The following sections work through this, using psfnormalize, which normalizes a UFO, with the option to convert between different UFO versions:
```
#!/usr/bin/env python3
'''Normalize a UFO and optionally convert between UFO2 and UFO3.
- If no options are chosen, the output font will simply be a normalized version of the font.'''
__url__ = 'https://github.com/silnrsi/pysilfont'
__copyright__ = 'Copyright (c) 2015 SIL International (https://www.sil.org)'
__license__ = 'Released under the MIT License (https://opensource.org/licenses/MIT)'
__author__ = 'David Raymond'
from silfont.core import execute
argspec = [
('ifont',{'help': 'Input font file'}, {'type': 'infont'}),
('ofont',{'help': 'Output font file','nargs': '?' }, {'type': 'outfont'}),
('-l','--log',{'help': 'Log file'}, {'type': 'outfile', 'def': '_conv.log'}),
('-v','--version',{'help': 'UFO version to convert to'},{})]
def doit(args) :
if args.version is not None : args.ifont.outparams['UFOversion'] = args.version
return args.ifont
def cmd() : execute("UFO",doit, argspec)
if __name__ == "__main__": cmd()
```
#### Header lines
Sample headers:
```
#!/usr/bin/env python3
'''Normalize a UFO and optionally convert between UFO2 and UFO3.
- If no options are chosen, the output font will simply be a normalized version of the font.'''
__url__ = 'https://github.com/silnrsi/pysilfont'
__copyright__ = 'Copyright (c) 2015 SIL International (https://www.sil.org)'
__license__ = 'Released under the MIT License (https://opensource.org/licenses/MIT)'
__author__ = 'David Raymond'
```
As well as providing the information for someone looking at the source file, the description comment (second line, which can be multi-line) is used by the framework when constructing the help text.
#### Import statement(s)
```
from silfont.core import execute
```
is required. Other imports from pysilfont or other libraries should be added, if needed.
#### Argument specification
The argument specifications take the form of a list of tuples, with one tuple per argument, eg:
```
argspec = [
('ifont',{'help': 'Input font file'}, {'type': 'infont'}),
('ofont',{'help': 'Output font file','nargs': '?' }, {'type': 'outfont'}),
('-l','--log',{'help': 'Log file'}, {'type': 'outfile', 'def': '_conv.log'}),
('-v','--version',{'help': 'UFO version to convert to'},{})]
```
Each argument has the format:
```
(argument name(s),argparse dict,framework dict)
```
argument name is either
- name for positional parameters, eg *‘ifont’*
- *-n, --name* or *--name* for other arguments, eg *‘-v’, ‘--version’*
**argparse dict** follows standard [argparse usage for .add_argument()](https://docs.python.org/2/library/argparse.html#the-add-argument-method). Help should always be included.
**framework dict** has optional values for:
- ‘type’ - the type of parameter, eg ‘outfile’
- ‘def’ - default for file names. Only applies if ‘type’ is a font or file.
- 'optlog' - For logs only. Flag to indicate the log file is optional - default False
‘Type’ can be one of:
| Value | Action |
|-------|-------------------------------------|
|infont|Open a font of that name and pass the font to the main function|
|outfont|If the main function to returns a font, save that to the supplied name|
|infile|Open a file for read and pass the file handle to the main function|
|incsv|Open a [csv](#support-for-csv-files) file for input and pass iterator to the main function|
|outfile|Open a file for writing and pass the file handle to the main function|
|filename|Filename to be passed as text|
|optiondict|Expects multiple values in the form name=val and passes a dictionary containing them|
If ‘def’ is supplied, the parameter value is passed through the [file name defaulting](#default-values-for-arguments) as specified below. Applies to all the above types except for optiondict.
In addition to options supplied in argspec, the framework adds [standard options](docs.md#standard-command-line-options), ie:
- -h, --help
- -q, --quiet
- -p, --params
- -l, --log
so these do not need to be included in argspec.
With -l, --log, this is still usually set in argspec to create default log file names. Set optlog to False if you want the log file to be optional.
#### doit() function
The main code of the script is in the doit() function.
The name is just by convention - it just needs to match what is passed to execute() at the end of the script. The
execute() function passes an args object to doit() containing:
- An entry for each command-line argument as appropriate, based on the full name of the argument
- eg with ``'-v','--version'``, args.version is set.
- Values are set for every entry in argspec, plus params, quiet and log added by the framework
- If no value is given on the command-line and the argument has no default then None is used.
- logger for the loggerobj()
- clarguments for a list of what was actually specified on the command line
- For parameters:
- params is a list of what parameters, if any, were specified on the command line
- paramsobj is the parameters object containing all [parameter](parameters.md) details
#### The final lines
These should always be:
```
def cmd() : execute(Tool,doit, argspec)
if __name__ == "__main__": cmd()
```
The first line defines the function that actually calls execute() to do the work, where Tool is one of:
- “UFO” to open fonts with pysilfont’s ufo.py module, returning a Ufont object
- “FP” to open fonts with fontParts, returning a font object
- “FT” to open fonts with FontTools, returning a TTfont object
- None if no font to be opened by execute()
- Other tools may be added in the future
The function must be called cmd(), since this is used by setup.py to install the commands.
The second line is the python way of saying, if you run this file as a script (rather than using it as a python module), execute the cmd() function.
Even if a script is initially just going to be used to be run manually, include these lines so no modification is needed to make it installable at a later date.
# Further framework notes
## Default values for arguments
Default values in [docs.md](docs.md#default-values) describes how file name defaulting works from a user perspective.
To set default values, either use the ‘default’ keyword in the argparse dict (for standard defaults) or the ‘def’ keyword in the framework dict to use Pysilfont’s file-name defaulting mechanism. Only one of these should be used. 'def' can't be used with the first positional parameter.
Note if you want a fixed file name, ie to bypass the file name defaulting mechanism, then use the argparse default keyword.
## Reporting
args.logger is a loggerobj(), and used to report messages to screen and log file. If no log file is set, messages are just to screen.
Messages are sent using
```
logger.log(<message text>, [severity level]>
```
Where severity level has a default value of W and can be set to one of:
- X Exception - For fatal programming errors
- S Severe - For fatal errors - eg input file missing
- E Errors - For serious errors that must be reported to screen
- P Progress - Progress messages
- W Warning - General warnings about anything not correct
- I Info - For more detailed reporting - eg the name of each glif file opened
- V Verbose - For even more messages!
Errors are reported to screen if the severity level is higher or equal to logger.scrlevel (default P) and to log based on loglevel (default W). The defaults for these can be set via parameters or within a script, if needed.
With X and S, the script is terminated. S should be used for user problems (eg file does not exist, font is invalid) and X for programming issues (eg an invalid value has been set by code). Exception errors are mainly used by the libraries and force a stack trace.
With Ufont objects, font.logger also points to the logger, but this is used primarily within the libraries rather than in scripts.
There would normally only be a single logger object used by a script.
### Changing reporting levels
loglevel and scrlevel *can* be set by scripts, but care should be taken not to override values set on the command line. To increase screen logging temporarily, use logger.raisescrlevel(<new level>) then set to previous value with logger.resetscrlevel(), eg
```
if not(args.quiet or "scrlevel" in params.sets["command line"]) :
logger.raisescrlevel("W") # Raise level to W if not already W or higher
<code>
logger.resetscrlevel()
```
### Error and warning counts
These are kept in logger.errorcount and logger.warningcount.
For scripts using the execute() framework, these counts are reported as progress messages when the script completes.
## Support for csv files
csv file support has been added to core.py with a csvreader() object (using the python csv module). In addition to the basic handling that the csv module provides, the following are supported:
- csvreader.firstline returns the first line of the file, so analyse headers if needed. Iteration still starts with the first line.
- Specifying the number of values expected (with minfields, maxfields, numfields)
- Comments (lines starting with #) are ignored
- Blank lines are also ignored
The csvreader() object is an iterator which returns the next line in the file after validating it against the min, max and num settings, if any, so the script does not have to do such validation. For example:
```
incsv = csvreader(<filespec>)
incsv.minfields = 2
Incsv.maxfields = 3
for line in inscv:
<code>
```
Will run `<code>` against each line in the file, skipping comments and blank lines. If any lines don’t have 2 or 3 fields, an error will be reported and the line skipped.
## Parameters
[Parameters.md](parameters.md) contains user, technical and developer’s notes on these.
## Chaining
With ufo.py scripts, core.py has a mechanism for chaining script function calls together to avoid writing a font to disk then reading it in again for the next call. In theory it could be used simply to call another script’s function from within a script.
This has not yet been used in practice, and will be documented (and perhaps debugged!) when there is a need, but there are example scripts to show how it was designed to work.
# pysilfont modules
These notes should be read in conjunction with looking at the comments in the code (and the code itself!).
## core.py
This is the main module that has the code to support:
- Reporting
- Logging
- The execute() function
- Chaining
- csvreader()
## etutil.py
Code to support xml handling based on xml.etree cElementTree objects. It covers the following:
- ETWriter() - a general purpose pretty-printer for outputting xml in a normalized form including
- Various controls on indenting
- inline elements
- Sorting attributes based on a supplied order
- Setting decimal precision for specific attributes
- doctype, comments and commentsafter
- xmlitem() class
- For reading and writing xml files
- Keeps record of original and final xml strings, so only needs to write to disk if changed
- ETelement() class
- For handling an ElementTree element
- For each tag in the element, ETelement[tag] returns a list of sub-elements with that tag
- process_attributes() processes the attributes of the element based on a supplied spec
- process_subelements() processes the subelements of the element based on a supplied spec
xmlitem() and ETelement() are mainly used as parent classes for other classes, eg in ufo.py.
The process functions validate the attributes/subelements against the spec. See code comments for details.
#### Immutable containers
Both xmlitem and ETelement objects are immutable containers, where
- object[name] can be used to reference items
- the object can be iterated over
- object.keys() returns a list of keys in the object
however, values can't be set with `object[name] = ... `; rather values need to be set using methods within child objects. For example, with a Uglif object, you can refer to the Uadvance object with glif['advance'], but to add a Uadvance object you need to use glif.addObject().
This is done so that values can be easily referenced and iterated over, but values can only be changed if appropriate methods have been defined.
Other Pysilfont objects also use such immutable containers.
## util.py
Module for general utilities. Currently just contains dirtree code.
#### dirTree
A dirTree() object represents all the directories and files in a directory tree and keeps track of the status of the directories/files in various ways. It was designed for use with ufo.py, so, after changes to the ufo, only files that had been added or changed were written to disk and files that were no longer part of the ufo were deleted. Could have other uses!
Each dirTreeItem() in the tree has details about the directory or file:
- type
- "d" or "f" to indicate directory or file
- dirtree
- For sub-directories, a dirtree() for the sub-directory
- read
- Item has been read by the script
- added
- Item has been added to dirtree, so does not exist on disk
- changed
- Item has been changed, so may need updating on disk
- towrite
- Item should be written out to disk
- written
- Item has been written to disk
- fileObject
- An object representing the file
- fileType
- The type of the file object
- flags
- Any other flags a script might need
## ufo.py
See [ufo.md](ufo.md) for details
## ftml.py
To be written
## comp.py
To be written
# Developer's notes
To cover items relevant to extending the library modules or adding new
To be written

17
docs/tests.md Normal file
View file

@ -0,0 +1,17 @@
# Test suite
pysilfont has a pytest-based test suite.
### install the test framework bits:
```
python3 -m pip install pytest
```
## set up the folders:
```
python3 tests/setuptestdata.py
```
## run the test suite:
```
pytest
```

159
docs/ufo.md Normal file
View file

@ -0,0 +1,159 @@
# Pysilfont - ufo support technical docs
# The Basics
UFO support is provided by the ufo.py library.
Most scripts work by reading a UFO into a Ufont object, making changes to it and writing it out again. The Ufont object contains many separate objects representing the UFO in a UFO 3 based hierarchy, even if the original UFO was format 2 - see [UFO 2 or UFO 3?](#ufo-2-or-ufo-3-) below.
Details of the [Ufont Object Model](#ufont-object-model) are given below, but in summary:
- There is an object for each file within the UFO (based on [xmlitem](technical.md#etutil.py))
- There is an object for each xml element within a parent object (based on [ETelement](technical.md#etutil.py))
- Data within objects can always(?) be accessed via generic methods based on the xml element tags
- For much data, there are object-specific methods to access data, which is easier than the generic methods
For example, a .glif file is:
- Read into a Uglif object which has:
- Methods for glyph-level data (eg name, format)
- objects for all the sub-elements within a glyph (eg advance, outline)
- Where an element type can only appear once in a glyph, eg advance, Uglif.*element-name* returns the relevant object
- Where an element can occur multiple times (eg anchor), Uglif.*element-name* returns a list of objects
- If an sub-element itself has sub-elements, then there are usually sub-element objects for that following the same pattern, eg Uglif.outline has lists of Ucontour and Ucomponent objects
It is planned that more object-specific methods will be added as needs arise, so raise in issue if you see specific needs that are likely to be useful in multiple scripts.
### UFO 2 or UFO 3?
The Ufont() object model UFO 3 based, so UFO 2 format fonts are converted to UFO 3 when read and then converted back to UFO 2 when written out to disk. Unless a script has code that is specific to a particular UFO format, scripts do not need to know the format of the font that was opened; rather they can just work in the UFO 3 format and leave the conversion to the library.
The main differences this makes in practice are:
- **Layers** The Ufont() always has layers. With UFO 2 fonts, there will be only one, and it can be accessed via Ufont.deflayer
- **Anchors** If a UFO 2 font uses the accepted practice of anchors being single point contours with a name and a type of "Move" then
- On reading the font, they will be removed from the list of contours and added to the list of anchors
- On writing the font, they will be added back into the list of contours
Despite being based on UFO 3 (for future-proofing), nearly all use of Pysilfont's UFO scripts has been with UFO 2-based projects so testing with UFO 3 has been minimal - and there are some [known limitations](docs.md#known-limitations).
# Ufont Object Model
A **Ufont** object represents the font using the following objects:
- font.**metainfo**: [Uplist](#uplist) object created from metainfo.plist
- font.**fontinfo**: [Uplist](#uplist) object created from fontinfo.plist, if present
- font.**groups**: [Uplist](#uplist) object created from groups.plist, if present
- font.**kerning**: [Uplist](#uplist) object created from kerning.plist, if present
- font.**lib**: [Uplist](#uplist) object created from lib.plist, if present
- self.**layercontents**: [Uplist](#uplist) object
- created from layercontents.plist for UFO 3 fonts
- synthesized based on the font's single layer for UFO 2 fonts
- font.**layers**: List of [Ulayer](#ulayer) objects, where each layer contains:
- layer.**contents**: [Uplist](#uplist) object created from contents.plist
- layer[**_glyph name_**]: A [Uglif](#uglif) object for each glyph in the font which contains:
- glif['**advance**']: Uadvance object
- glif['**unicode**']: List of Uunicode objects
- glif['**note**']: Unote object (UFO 3 only)
- glif['**image**']: Uimage object (UFO 3 only)
- glif['**guideline**']: List of Uguideline objects (UFO 3 only)
- glif['**anchor**']: List of Uanchor objects
- glif['**outline**']: Uoutline object which contains
- outline.**contours**: List of Ucontour objects
- outline.**components**: List of Ucomponent objects
- glif['**lib**']: Ulib object
- font.**features**: UfeatureFile created from features.fea, if present
## General Notes
Except for UfeatureFile (and Ufont itself), all the above objects are set up as [immutable containers](technical.md#immutable-containers), though the contents, if any, depend on the particular object.
Objects usually have a link back to their parent object, eg glif.layer points to the Ulayer object containing that glif.
## Specific classes
**Note - the sections below don't list all the class details** so also look in the code in ufo.py if you need something not listed - it might be there!
### Ufont
In addition to the objects listed above, a Ufont object contains:
- self.params: The [parameters](parameters.md) object for the font
- self.paramsset: The parameter set within self.params specific to the font
- self.logger: The logger object for the script
- self.ufodir: Text string of the UFO location
- self.UFOversion: from formatVersion in metainfo.plist
- self.dtree: [dirTree](technical.md#dirtree) object representing all the files on fisk and their status
- self.outparams: The output parameters for the font, initially set from self.paramset
- self.deflayer:
- The single layer for UFO 2 fonts
- The layer called public.default for UFO 3 fonts
When creating a new Ufont() object in a script, it is normal to pass args.paramsobj for params so that it has all the settings for parameters and logging.
self.write(outputdir) will write the UFO to disk. For basic scripts this will usually be done by the execute() function - see [writing scripts](technical.md#writing-scripts).
self.addfile(type) will add an empty entry for any of the optional plist files (fontinfo, groups, kerning or lib).
When writing to disk, the UFO is always normalized, and only changed files will actually be written to disk. The format for normalization, as well as the output UFO version, are controlled by values in self.outparams.
### Uplist
Used to represent any .plist file, as listed above.
For each key,value pair in the file, self[key] contains a list:
- self[key][0] is an elementtree element for the key
- self[key][1] is an elementtree element for the value
self.getval(key) will return:
- the value, if the value type is integer, real or string
- a list if the value type is array
- a dict if the value type is dict
- None, for other value types (which would only occur in lib.plist)
- It will throw an exception if the key does not exist
- for dict and array, it will recursively process dict and/or array subelements
Methods are available for adding, changing and deleting values - see class \_plist in ufo.py for details.
self.font points to the parent Ufont object
### Ulayer
Represents a layer in the font. With UFO 2 fonts, a single layer is synthesized from the glifs folder.
For each glyph, layer[glyphname] returns a Uglif object for the glyph. It has addGlyph and delGlyph functions.
### Uglif
Represents a glyph within a layer. It has child objects, as listed below, and functions self.add and self.remove for adding and removing them. For UFO 2 fonts, and contours identified as anchors will have been removed from Uoutline and added as Uanchor objects.
#### glif child objects
There are 8 child objects for a glif:
| Name | Notes | UFO 2 | Multi |
| ---- | -------------------------------- | --- | --- |
| Uadvance | Has width & height attributes | Y | |
| Uunicode | Has hex attribute | Y | Y |
| Uoutline | | Y | |
| Ulib | | Y | |
| Unote | | | |
| Uimage | | | |
| Uguideline | | | Y |
| Uanchor | | | Y |
They all have separate object classes, but currently (except for setting attributes), only Uoutline and Ulib have any extra code - though more will be added in the future.
(With **Uanchor**, the conversion between UFO 3 anchors and the UFO 2 way of handling anchors is handled by code in Uglif and Ucontour)
**Ulib** shares a parent class (\_plist) with [Uplist](#uplist) so has the same functionality for managing key,value pairs.
#### Uoutline
This has Ucomponent and Ucontour child objects, with addobject, appendobject and insertobject methods for managing them.
With Ucontour, self['point'] returns a list of the point subelements within the contour, and points can be managed using the methods in Ulelement. other than that, changes need to be made by changing the elements using elementtree methods.
# Module Developer Notes
To be written