WASMFS and WASI support

[petersalomonsen]

Work in progress. Here's a screenshot of a Wasi build, displaying the available commands of lg2.wasm running with wasmtime:

And here's an initial test with WASMFS:

844fa31

[kesavkolla]

@petersalomonsen Any update on this? Is there any effort pending for this to get merged?

[petersalomonsen]

@petersalomonsen Any update on this? Is there any effort pending for this to get merged?

@kesavkolla I will not merge this, but rather split it up into separate parts for what's in here:

• WASMFS/OPFS: From what I learned [here])(WasmFS: Current directory ignores custom backend emscripten-core/emscripten#18112) either threads or JSPI is needed, which are currently not supported without enabling special flags in chrome. For wasm-git it should be a special build for those who are interested in it, and it also makes sense to have it in a separate PR.
• WASI: I got quickly into limitations with WASI 1, especially when it comes to network requests for pull/push. I have done some experimenting on the component model (WASI 2), and so I will rather revisit that when it becomes more official. Also in a separate PR then.
• ES6 build is probably the lowest hanging fruit here, so that will come soon
• Devcontainer setup is also straightforward, so I will separate that out

Which of these are you most interested in?

[kesavkolla]

@petersalomonsen Any update on this? Is there any effort pending for this to get merged?

@kesavkolla I will not merge this, but rather split it up into separate parts for what's in here:

• WASMFS/OPFS: From what I learned [here])(WasmFS: Current directory ignores custom backend emscripten-core/emscripten#18112) either threads or JSPI is needed, which are currently not supported without enabling special flags in chrome. For wasm-git it should be a special build for those who are interested in it, and it also makes sense to have it in a separate PR.
• WASI: I got quickly into limitations with WASI 1, especially when it comes to network requests for pull/push. I have done some experimenting on the component model (WASI 2), and so I will rather revisit that when it becomes more official. Also in a separate PR then.
• ES6 build is probably the lowest hanging fruit here, so that will come soon
• Devcontainer setup is also straightforward, so I will separate that out

Which of these are you most interested in?

I don't know what is the best behavior either WASI or WASMFS/OPFS. Ideally the requirement is to run the git commands right from the browser's filesystem. I was hoping OPFS is the right choice as I can then use the filesystem API to access the files and then once changes are made use git command to push the changes.

[petersalomonsen]

@petersalomonsen Any update on this? Is there any effort pending for this to get merged?

@kesavkolla I will not merge this, but rather split it up into separate parts for what's in here:

• WASMFS/OPFS: From what I learned [here])(WasmFS: Current directory ignores custom backend emscripten-core/emscripten#18112) either threads or JSPI is needed, which are currently not supported without enabling special flags in chrome. For wasm-git it should be a special build for those who are interested in it, and it also makes sense to have it in a separate PR.
• WASI: I got quickly into limitations with WASI 1, especially when it comes to network requests for pull/push. I have done some experimenting on the component model (WASI 2), and so I will rather revisit that when it becomes more official. Also in a separate PR then.
• ES6 build is probably the lowest hanging fruit here, so that will come soon
• Devcontainer setup is also straightforward, so I will separate that out

Which of these are you most interested in?

I don't know what is the best behavior either WASI or WASMFS/OPFS. Ideally the requirement is to run the git commands right from the browser's filesystem. I was hoping OPFS is the right choice as I can then use the filesystem API to access the files and then once changes are made use git command to push the changes.

All right, yes I think that also OPFS should be possible to implement now that WasmFS seems to have come quite far, and the only requirement will then be that JSPI is enabled. Soon it may also be enabled by default in the browsers. Will create a separate PR for OPFS/WasmFS soon.

[antoniogagos]

I'm encountering an issue with mounting OPFS.
The following code hangs indefinitely at FS.mount(OPFS, {}, '/test'); when using threads:

console.log('Creating /test directory...');
FS.mkdir('/test');
console.log('Mounting OPFS at /test...');
FS.mount(OPFS, {}, '/test'); // Hangs here
console.log('Mounted OPFS successfully');
console.log(FS.readdir('/test'));

I tried with:

• Emscripten 3.1.74
• Chrome 133 (without enabling JSPI)
• Running in a worker with SharedArrayBuffer correctly enabled.
• Build config with: -pthread -s PROXY_TO_PTHREAD=1 -s WASMFS=1 -lopfs.js -s FORCE_FILESYSTEM=1

I also tried excluding ASYNCIFY since it shouldn't be necessary with threads AFAIK.
Is there a way to make this work before JSPI is available?

[antoniogagos]

Compiling with a minimum pool size greater than 0 (-s PTHREAD_POOL_SIZE=1) fixed the issue. Without this setting, no worker threads were being created.

I can now manipulate OPFS using FS, but the interaction with libgit2 seems to be broken.

FS.mkdir('/opfs');
FS.mount(OPFS, {}, '/opfs');
FS.chdir('/opfs');
FS.readdir('.'); // works and prints existing content at OPFS

FS.chdir('/opfs/my-repo'); // location of an existing valid git repo

lg.callMain(['status']); // fails - can't not find a repository

There appears to be a conflict with path resolution. Removing the leading slashes from paths added by git_fs_path_prettify partially resolves the issue: repositories are detected, but other operations like init fail (it creates the .git folder but fails to create anything else because it can't find the folder)