Currently fpm cli only works against local file system. We want to now evolve into something that can both use local file system, and work off a database.

`fpm serve` is “dynamic”
`fpm serve` runs an HTTP server, and serves the FTD and other files in current fpm package dynamically. Contrast it with `fpm build && cd .build && python http.server`, when running like this we always get the same content, since `fpm build` runs once, and builds all the files in `.build` folder.

Now it’s a static file, and `python http.server` always serves the same content for same URL. `fpm serve` does this on the fly, when the HTTP request comes. Which means if the ftd file is dynamic, say ftd file uses a processor that fetches data from database, such a file would refetch the data on every HTTP request when served via `fpm serve` but when `fpm build` is used, at the time of build the data would be fetched.

Hosting `fpm serve` With Files
Since `fpm serve` is dynamic, it offers authors write more powerful documents, they can use `ftd/fpm` better. But `spm serve` so far is only running locally, on your machine, exposing the server on `http://127.0.0.1:8000`.

You can run `fpm serve` on your own VPS on Digital Ocean or EC2. You will have to checkout your FPM package content on the VPS.

For this we do not need anything more, the moment `fpm serve` dynamically PR is merged we are good to go.

But it has one main disadvantage: managing FTD files would still have to be done why `sFTP/FTP/git` etc. When the fpm package content changes, you will have to deploy the content to your server somehow.

FPM Package Digest: `digest.json`
`digest.json` is an in memory data structure which is generated first when any `fpm` command is invoked locally. Content of `digest.json` is also stored in `packages` table on remote.

For every FTD and markdown file in the package, it will contain the content of that file.

For every non FTD/md file it will contain the filename, which will be stored as a list in `other-files` key.

It will also contain history and tracking metadata.

digest.json

{
    "index.ftd": "content of index.ftd",
    "other-files": [
        "images/logo.png",
        "foo.md",
        "hello.py"
    ],
    "history": {
        "index.ftd": [
            { "updated": "<unix-timestamp-nanoseconds>" },
            { "created": "<unix-timestamp-nanoseconds>" }
        ]
    },
    "tracks": {
        "index.ftd": [
            {
                "foo.ftd": {
                    "last-merged-on": "<timestamp of foo.ftd>"
                }
            }
        ]
    }
}

{
    "index.ftd": "content of index.ftd",
    "other-files": [
        "images/logo.png",
        "foo.md",
        "hello.py"
    ],
    "history": {
        "index.ftd": [
            { "updated": "<unix-timestamp-nanoseconds>" },
            { "created": "<unix-timestamp-nanoseconds>" }
        ]
    },
    "tracks": {
        "index.ftd": [
            {
                "foo.ftd": {
                    "last-merged-on": "<timestamp of foo.ftd>"
                }
            }
        ]
    }
}

NOTE: we are showing timestamps as “string”, but it will be actually be integer.

When we download a package, we will first extract it’s content and create .packages/<package-name>.digest.json.

What’s In DB?

`other-files` table
For every “other-file”, we will have a row containing the content of that file.

select * from other_files;
| filename              | content of file       |
| amitu.com/foo.png     | <content of foo.png>  |
| fifthtry.com/logo.png | <content of logo.png> |

select * from other_files;
| filename              | content of file       |
| amitu.com/foo.png     | <content of foo.png>  |
| fifthtry.com/logo.png | <content of logo.png> |

`packages` table
For every package, we will also have a row containing the package.digest.json for all packages.

select * from packages;
| package name        | content of digest.json                    |
+---------------------+-------------------------------------------|
| amitu.com           | <content of digest.json for amitu.com>    |
| amitu.com/x         | <content of digest.json for amitu.com>    |
| amitu.com/x/y       | <content of digest.json for amitu.com>    |
| fifthtry.com        | <content of digest.json for fifthtry.com> |

select * from packages;
| package name        | content of digest.json                    |
+---------------------+-------------------------------------------|
| amitu.com           | <content of digest.json for amitu.com>    |
| amitu.com/x         | <content of digest.json for amitu.com>    |
| amitu.com/x/y       | <content of digest.json for amitu.com>    |
| fifthtry.com        | <content of digest.json for fifthtry.com> |

`history` table
The history will be stored in history table:

select * from history;
| filename            | timestamp        | event   | content   |
| amitu.com/index.ftd | <unix-timestamp> | created | <content> |
| amitu.com/index.ftd | <unix-timestamp> | updated | <content> |

select * from history;
| filename            | timestamp        | event   | content   |
| amitu.com/index.ftd | <unix-timestamp> | created | <content> |
| amitu.com/index.ftd | <unix-timestamp> | updated | <content> |

In future we will also store the diff column, which will show the difference in this document with respect to previous row.

And also cr column, which will contain if this change came from a cr, if so the cr number, if change happened directly on main, cr will be null.

`fpm-files` table
Since FPM.ftd file is needed to serve every static file (because FPM.ftd contains authentication/access control information), we have read it very frequently, so we will keep this in a separate table only to be used when serving static file requests.

select * from fpm-files;

| package name | FPM.ftd content         |
+--------------+-------------------------+
| amitu.com    | <amitu/FPM.ftd content> |

select * from fpm-files;

| package name | FPM.ftd content         |
+--------------+-------------------------+
| amitu.com    | <amitu/FPM.ftd content> |

How To Load .digest.json On Local?
Look for `.fpm-workspace/digest.json`.

How To Load .digest.json On Remote?
Say if db looks like this:

select * from packages;
| package              | content                                   |
+----------------------+-------------------------------------------+
| amitu.com/           | <content of digest.json for amitu.com>    |
| amitu.com/x/         | <content of digest.json for amitu.com>    |
| amitu.com/x/y/       | <content of digest.json for amitu.com>    |
| fifthtry.com/        | <content of digest.json for fifthtry.com> |

select * from packages;
| package              | content                                   |
+----------------------+-------------------------------------------+
| amitu.com/           | <content of digest.json for amitu.com>    |
| amitu.com/x/         | <content of digest.json for amitu.com>    |
| amitu.com/x/y/       | <content of digest.json for amitu.com>    |
| fifthtry.com/        | <content of digest.json for fifthtry.com> |

And a request to amitu.com/foo comes, we have to read the content of amitu.com row. But if amitu/x/y/z comes then we have to read of content of amitu.com/x/y. How do we do know which row to read?

Step 1: find the domain name only, amitu.com, and then look for all rows:

select package from packages where package ilike "amitu.com/%"

| amitu.com/     |
| amitu.com/x/   |
| amitu.com/x/y/ |

select package from packages where package ilike "amitu.com/%"

| amitu.com/     |
| amitu.com/x/   |
| amitu.com/x/y/ |

Step 2: Find the largest package from this list, where "amitu.com/foo".starts_with(package) is true. Here the answer would be amitu.com/, so this is the package that contains amitu.com/foo.

Now select content from packages where package=amitu.com.

How to serve the file?
A request has come to fpm http server, we want to serve it.

FTD files: URL with no extension, or URL ending with `index.html`
If URL ends with index.html, we will delete the ending index.html to get the `real path`.

If the `real path` is present in `digets` json for the package, eg if the digest was:

{
    "foo/index.ftd": "<content>",
    "FPM.ftd": "<content>"
    ... other stuff omitted ...
}

{
    "foo/index.ftd": "<content>",
    "FPM.ftd": "<content>"
    ... other stuff omitted ...
}

NOTE: if the URL contains - then the real path would be the part till the first -, eg if path was amitu.com/foo/-/bar/, the real path would be foo. Ask Arpita how to handle such URLs.

If the real path is foo, then we look for <real path.ftd> (foo.ftd) or <real path>/index.ftd (foo/index.ftd) has to be present.

We know it is a ftd file and we can serve it. When “process_ftd() is called, on every import (ftd::Interpreter::StuckOnImport) corresponding to a foreign package we have to read the row for that package from packagestable (or from disc,.packages/.digest.json` if running locally). Here we do not have to do step 1, because the list of dependencies for this package already known, so we have to do step 1 equivalent from content of FPM.ftd only, we do not have to do additional (db/fs) read.

If the file is missing in digest.json then we give 404.

Any other URL (non ftd url, mostly images)
We assume this is static file, so we only do the step 1, and not step 2 as we are not going to need the full digest. We will read FPM.ftd content from `fpm-files` for the package name we found from step 1. Using FPM.ftd we will check if auth/acl allows you to read the static file. If not we give 403.

If acl works, we then do a query on the `other-files` table, to get the content of that file and serve it.

How do we handle markdown files?
Markdown files (and their content) would also be present in `digest.json` file. When looking for `foo` we will look for `foo.ftd`, `foo/index.ftd`, `foo.md` and `foo/README.md`, in this order.

On local, how often do we regenerate `digest.json`?
When we lauch `fpm serve`, it first reads the content of current package directory, and generates an in-memory `digest.json` and starts a background thread which watches for file system changes, and keep updating the in-memory `digest.json`.

What about local changes in dependencies?
Since people may modify dependencies as part of development, we have to generate `digest.json` when any fpm command starts. The file system watcher of FPM serve will also watch the dependencies, and update their `digest.json` as well.

Note: local changes in dependencies will make output of subsequent fpm commands unreliable. The diff may be wrong if you are tracking things, `fpm status` may give wrong info etc etc. In future we will use “fallback to ..” technique, so if you want to develop a dependency, you do not modify it in `.packages` folder, but checkout the dependency in `..` of current package, and our dependency checker will look both in `.packages` and `..` when its looking for `foo.com/bar` package as a dependency of current package.

`digest.json` on remote
Is only updated when someone does `fpm sync` and any of the content on remote changes. Since the digest contains history, meta data for all static file, so if any file ever changes, `digest.json` will have to be updated.

FPM New Design

What’s In DB?

other-files tableFor every “other-file”, we will have a row containing the content of that file.

packages tableFor every package, we will also have a row containing the package.digest.json for all packages.

history tableThe history will be stored in history table:

fpm-files tableSince FPM.ftd file is needed to serve every static file (because FPM.ftd contains authentication/access control information), we have read it very frequently, so we will keep this in a separate table only to be used when serving static file requests.

How To Load .digest.json On Local?Look for .fpm-workspace/digest.json.

How To Load .digest.json On Remote?Say if db looks like this:

How to serve the file?A request has come to fpm http server, we want to serve it.

FTD files: URL with no extension, or URL ending with index.htmlIf URL ends with index.html, we will delete the ending index.html to get the real path. If the real path is present in digets json for the package, eg if the digest was:

How do we handle markdown files?Markdown files (and their content) would also be present in digest.json file. When looking for foo we will look for foo.ftd, foo/index.ftd, foo.md and foo/README.md, in this order.

On local, how often do we regenerate digest.json?When we lauch fpm serve, it first reads the content of current package directory, and generates an in-memory digest.json and starts a background thread which watches for file system changes, and keep updating the in-memory digest.json.

digest.json on remoteIs only updated when someone does fpm sync and any of the content on remote changes. Since the digest contains history, meta data for all static file, so if any file ever changes, digest.json will have to be updated.