golang-forgejo-f3-gof3/README.md

## gof3

As a CLI or as a library, GoF3 provides a single operation: mirroring. The origin and destination are designated by the URL of a forge and a path to the resource. For instance, `mirror --from-type forgejo --from https://code.forgejo.org/forgejo/lxc-helpers --to-type F3 --to /some/directory` will mirror a project in a local directory using the F3 format.

## Building

* Install go >= v1.21
* make f3-cli
* ./f3-cli mirror -h

## Example

### To F3

Login to https://code.forgejo.org and obtain an application token with
read permissions at https://code.forgejo.org/user/settings/applications.

```sh
f3-cli mirror \
  --from-type forgejo --from-forgejo-url https://code.forgejo.org \
  --from-forgejo-token $codetoken \
  --from-path /forge/organizations/actions/projects/cascading-pr \
  --to-type filesystem --to-filesystem-directory /tmp/cascading-pr
```

### From F3

Run a local Forgejo instance with `serials=1 tests/setup-forgejo.sh` and obtain
an application token with:

```sh
docker exec --user 1000 forgejo1 forgejo admin user generate-access-token -u root --raw --scopes 'all,sudo'
```

Mirror issues

```sh
f3-cli mirror \
  --from-type filesystem --from-filesystem-directory /tmp/cascading-pr \
  --from-path /forge/organizations/actions/projects/cascading-pr/issues \
  --to-type forgejo --to-forgejo-url http://0.0.0.0:3001 \
  --to-forgejo-token $localtoken
```

Visit them at http://0.0.0.0:3001/actions/cascading-pr/issues

## Testing

### Requirements

The tests require a live GitLab instance as well as a live Forgejo instance and will use up to 16GB of RAM.

* Install docker
* `./test/run.sh`

## License

This project is [MIT licensed](LICENSE).

## Architecture

[F3](https://f3.forgefriends.org/) is a hierarchy designed to be stored in a file system. It is represented in memory with the [tree/generic](tree/generic) abstract data structure that can be saved and loaded from disk by the [forges/filesystem](forges/filesystem) driver. Each forge (e.g. [forges/forgejo](forges/forgejo)) is supported by a driver that is responsible for the interactions of each resource (e.g `issues`, `asset`, etc.).

### Tree

[tree/f3](tree/f3) implements a [F3](https://f3.forgefriends.org/) hierarchy based on the [tree/generic](tree/generic) data structure. The [tree](tree/generic/tree.go) has a [logger](logger) for messages, [options](options) defining which forge it relates to and how and a pointer to the root [node](tree/generic/node.go) of the hierarchy (i.e. the `forge` F3 resource).

The node ([tree/generic/node.go](tree/generic/node.go)) has:

* a unique id (e.g. the numerical id of an `issue`)
* a parent
* chidren (e.g. `issues` children are `issues`, `issue` children are `comments` and `reactions`)
* a kind that maps to a F3 resource (e.g. `issue`, etc.)
* a driver for its concrete implementation for a given forge

It relies on a forge driver for the concrete implemenation of a F3 resource (issue, reaction, repository, etc.). For instance the `issues` driver for Forgejo is responsible for listing the existing issues and the `issue` driver is responsible for creating, updating or deleting a Forgejo issue.

### F3 archive

The [F3 JSON schemas](https://code.forgejo.org/f3/f3-schemas/-/tree/main) are copied in [f3/schemas](f3/schemas). Their internal representation and validation is found in a source file named after the resource (e.g. an `issue` represented by [f3/schemas/issue.json](f3/schemas/issue.json) is implemented by [f3/issue.go](f3/issue.go)).

When a F3 resource includes data external to the JSON file (i.e. a Git repository or an asset file), the internal representation has a function to copy the data to the destination given in argument. For instance:

* [f3/repository.go](f3/repository.go) `FetchFunc(destination)` will `git fetch --mirror` the repository to the `destination` directory.
* [f3/releaseasset.go](f3/releaseasset.go) `DownloadFunc()` returns a `io.ReadCloser` that will be used by the caller to copy the asset to its destination.

### Options

The Forge options at [options/interface.go](options/interface.go) define the parameters given when a forge is created:

Each forge driver is responsible for registering the options (e.g. [Forgejo options](forges/forgejo/options/options.go)) and for registering a factory that will create these options (e.g. [Forgejo options registration](forgejo/main.go)). In addition to the options that are shared by all forges such as the logger, it may define additional options.

### Driver interface

For each [F3](https://f3.forgefriends.org/) resource, the driver is responsible for:

* copying the [f3](f3) argument to `FromFormat` to the forge
* `ToFormat` reads from the forge and convert the data into an [f3/resources.go](f3/resources.go)

A driver must have a unique name (e.g. `forgejo`) and [register](forges/forgejo/main.go):

* an [options factory](options/factory.go)
* a [forge factory](tree/f3/forge_factory.go)

#### Tree driver

The [tree driver](tree/generic/driver_tree.go) functions (e.g. [forges/forgejo/tree.go](forges/forgejo/tree.go)) specialize [NullTreeDriver](tree/generic/driver_tree.go).

* **Factory(ctx context.Context, kind generic.Kind) generic.NodeDriverInterface** creates a new node driver for a given [`Kind`](tree/f3/kind.go).
* **GetPageSize() int** returns the default page size.

#### Node driver

The [node driver](tree/generic/driver_node.go) functions for [each `Kind`](tree/f3/kind.go) (e.g. `issues`, `issue`, etc.)  specialize [NullNodeDriver](tree/generic/driver_node.go). The examples are given for the Forgejo [`issue`](forges/forgejo/issue.go) and [`issues`](forges/forgejo/issues.go) drivers, matching the REST API endpoint to the driver function.

* **ListPage(context.Context, page int) ChildrenSlice** returns children of the node paginated  [GET /repos/{owner}/{repo}/issues](https://code.forgejo.org/api/swagger/#/issue/issueListIssues)
* **Get(context.Context)** get the content of the resource (e.g. [GET /repos/{owner}/{repo}/issues/{index}](https://code.forgejo.org/api/swagger/#/issue/issueGetIssue))
* **Put(context.Context) NodeID** create a new resource and return the identifier (e.g. [POST /repos/{owner}/{repo}/issues](https://code.forgejo.org/api/swagger/#/issue/issueCreateIssue))
* **Patch(context.Context)** modify an existing resource (e.g. [PATCH /repos/{owner}/{repo}/issues/{index}](https://code.forgejo.org/api/swagger/#/issue/issueEditIssue))
* **Delete(context.Context)** delete an existing resource (e.g. [DELETE /repos/{owner}/{repo}/issues/{index}](https://code.forgejo.org/api/swagger/#/issue/issueDelete))
* **NewFormat() f3.Interface** create a new `issue` F3 object
* **FromFormat(f3.Interface)** set the internal representation from the given F3 resource
* **ToFormat() f3.Interface** convert the internal representation into the corresponding F3 resource. For instance the internal representation of an `issue` for the Forgejo driver is the `Issue` struct of the Forgejo SDK.

#### Options

The [options](options) created by the factory are expected to provide the [options interfaces](options/interface.go):

* Required
  * LoggerInterface
  * URLInterface
* Optional
  * CLIInterface if additional CLI arguments specific to the forge are supported

For instance [forges/forgejo/options/options.go](forges/forgejo/options/options.go) is created by [forges/forgejo/options.go](forges/forgejo/options.go).

### Driver implementation

A driver for a forge must be self contained in a directory (e.g. [forges/forgejo](forges/forgejo)). Functions shared by multiple forges are grouped in the [forges/helpers](forges/helpers) directory and split into one directory per `Kind` (e.g. [forges/helpers/pullrequest](forges/helpers/pullrequest)).

* [options.go](forges/forgejo/options.go) defines the name of the forge in the Name variable (e.g. Name = "forgejo")
* [options/options.go](forges/forgejo/options/options.go) defines the options specific to the forge and the corresponding CLI flags
* [main.go](forges/forgejo/main.go) calls f3_tree.RegisterForgeFactory to create the forge given its name
* [tree.go](forges/forgejo/tree.go) has the `Factory()` function that maps a node kind (`issue`, `reaction`, etc.) into an object that is capable of interacting with it (CRUD).
* one file per `Kind` (e.g. [forges/forgejo/issues.go](forges/forgejo/issues.go)).

### Idempotency

Mirroring is idempotent: it will produce the same result if repeated multiple times. The drivers functions are not required to be idempotent.

* The `Put` function will only be called if the resource does not already exist.
* The `Patch` and `Delete` functions will only be called if the resource exists.

### Identifiers mapping

When a forge (e.g. Forgejo) is mirrored on the filesystem, the identifiers are preserved verbatim (e.g. the `issue` identifier). When the filesystem is mirrored to a forge, the identifiers cannot always be preserved. For instance if an `issue` with the identifier 1234 is downloaded from Forgejo and created on another Forgejo instance, it will be allocated an identifier by the Forgejo instance. It cannot request to be given a specific identifier.

### References

A F3 resource may reference another F3 resource by a path. For instance the user that authored an issue is represented by `/forge/users/1234` where `1234` is the unique identifier of the user. The reference is relative to the forge. The mirroring of a forge to another is responsible for converting the references using the identifier mapping stored in the origin forge. For instance if `/forge/users/1234` stored in the filesystem is created in Forgejo as `/forge/users/58`, the `issue` stored in the filesystem with its authored as `/forge/users/1234` will be created in Forgejo to be authored by `/forge/users/58` instead.

### Logger

The [tree/generic](tree/generic) has a pointer to a logger implementing [logger.Interface](logger/interface.go) which is made available to the nodes and the drivers.

### Context

All functions except for setters and getters have a `context.Context` argument which is checked (using [util/terminate.go](util/terminate.go)) to not be `Done` before performing a long lasting operation (e.g. a REST API call or a call to the Git CLI). It is not used otherwise.

### Error model

When an error that cannot be recovered from happens, `panic` is called, otherwise an `Error` is logged.

### CLI

The CLI is in [cmd](cmd) and relies on [options](options) to figure out which options are to be implemented for each supported forge.

## Hacking

### Local tests

The forge instance is deleted before each run and left running for forensic analysis when the run completes.

```sh
./tests/run.sh test_forgejo # http://0.0.0.0:3001 user root, password admin1234
./tests/run.sh test_gitlab # http://0.0.0.0:8181 user root, password Wrobyak4
./tests/run.sh test_gitea # http://0.0.0.0:3001 user root, password admin1234
```

Restart a new forge with:

```sh
./tests/run.sh run_forgejo # http://0.0.0.0:3001 user root, password admin1234
./tests/run.sh run_gitlab # http://0.0.0.0:8181 user root, password Wrobyak4
./tests/run.sh run_gitea # http://0.0.0.0:3001 user root, password admin1234
```

The compliance test resources are deleted, except if the environment variable `GOF3_TEST_COMPLIANCE_CLEANUP=false`.

```sh
GOF3_TEST_COMPLIANCE_CLEANUP=false GOF3_FORGEJO_HOST_PORT=0.0.0.0:3001 go test -run=TestF3Forge/forgejo -v code.forgejo.org/f3/gof3/...
```

### Code coverage

```sh
export SCRATCHDIR=/tmp/gof3
./tests/run.sh # collect coverage for every test
./tests/run.sh run_forgejo # update coverage for forgejo
./tests/run.sh test_merge_coverage # merge coverage from every test
go tool cover -func /tmp/gof3/merged.out # show coverage per function
uncover /tmp/gof3/merged.out GeneratorSetReviewComment # show which lines of the GeneratorSetReviewComment function are not covered
```

### F3 schemas

The JSON schemas come from [the f3-schemas repository](https://code.forgejo.org/f3/f3-schemas) and
should be updated as follows:

```
cd f3 ; rm -fr schemas ; git --work-tree schemas clone https://code.forgejo.org/f3/f3-schemas ; rm -fr f3-schemas schemas/.gitignore schemas/.forgejo
```

## Funding

See the page dedicated to funding in the [F3 documentation](https://f3.forgefriends.org/funding.html)