CLI and networking support #119

adamziel · 2023-01-25T11:33:04Z

Description

Adds support for CLI SAPI and networking in node.js:

> npm run build

> node ./build-cli/php-cli.js -r 'echo "Hello from PHP !";'
Hello from PHP !

> node ./build-cli/php-cli.js -r 'echo substr(file_get_contents("https://wordpress.org"), 0, 16);'
<!DOCTYPE html>

> node ./build-cli/php-cli.js -r 'echo phpversion();'
8.2.0-dev

> PHP=5.6 node ./build-cli/php-cli.js -r 'echo phpversion();'
5.6.40

Highlights:

Networking is supported (including MySQL and HTTPS)
Switching PHP versions is supported.
Most WordPress PHPUnit tests pass. The failures are caused by missing extensions and a few misconfigured settings
PHP Interactive mode is supported but the arrow keys don't work
wp-cli works

In broad strokes:

CLI SAPI is compiled with libedit (readline replacement) and ncurses.
Network calls are asynchronous. Emscripten's Asyncify enables calling asynchronous code from synchronous code. TCP sockets are shimmed with a WebSocket connection to a built-in proxy server running on localhost. It supports data transfer, arbitrary connection targets, and setting a few TCP socket options.
PHP's OpenSSL uses the same CA certs as Node.js
PHP 5.6 is ed to work with OpenSSL 1.1.0 and many other small es are introduced. For more details, see es overview, Dockerfile, and phpwasm-emscripten-library.js

Future work:

PHP Interactive server isn't supported yet. Adding support is a matter of making the incoming connection polling non-blocking using Asyncify.
Use a more recent OpenSSL version
Better support for CLI interactive mode

dmsnell · 2023-01-27T20:34:46Z

Is there a discussion of where these es come from or why they are what they are? Is there a reason they have to be here vs. in a fork of the upstream projects? I saw es to SSL, which you know, catches the eye.

eliot-akira · 2023-01-28T11:28:19Z

Here's some discussion started about the es: Propose PRs for the projects that are now ed ad-hoc.

adamziel · 2023-01-30T10:19:53Z

Is there a discussion of where these es come from or why they are what they are? Is there a reason they have to be here vs. in a fork of the upstream projects? I saw es to SSL, which you know, catches the eye.

As @eliot-akira mentioned, I'd love to merge these es to the upstream projects. I will open Pull Requests for actively maintained projects like Emscripten. Unfortunately, PHP 5.6 and some friends are not getting new releases anymore. The OpenSSL you've mentioned is actually a PHP 5.6 to add OpenSSL 1.1.0 support. It's been proposed for upstream PHP 5.6 and rejected. AFAIK most webhosts who still support PHP 5.6 are maintaining their own fork to keep applying security es.

Here's the overview of all the es:

This PR removes all sqlite3 es originally inherited from https://.com/seanmorris/php-wasm. These es moved SQLite3 sources directly to php-src, updated the #include paths to bake SQLite3 into the PHP build, and prevented a sqlite3 > 3.7.4 version check during the PHP ./configure command. This PR instead builds the correct sqlite3 version as a static WASM library using Emscripten's SIDE_MODULE and lets the linker connect the dots.
ncurses. – PHP CLI is compiled with readline support and linked with ncurses, however building ncurses using Emscripten fails since it involved creating some database files using the make_hash and make_keys binaries that can't be compiled as WASM. The ensures these databases are built using the system-native versions of make_hash and make_keys. Source 1 Source 2 (also provided inline).
bison2.7. – Compatibility with Emscripten's version of glibc which is newer than the one supported by Bison 2.7. Source: OpenWRT, more details (also provided inline).
php5.6-openssl1.1 – Adds OpenSSL 1.1.0 support for PHP 5.6. Source (also provided inline).
php5.6. – Adds a missing header declaration, otherwise emscripten won't compile the mbstring extension. Source: me.
php7.0. up to php8.0.:
- Replaces zend_ulong with size_t, zend_off_t with long long, and ensures a pointer is used when calling php_stream_seek. It solves Esmcripten compilation errors, but I don't have a good grasp why is this even needed. Ideally this part of the could be removed. Source: me.
- Avoids copying empty files as that crashes Emscripten's runtime. That's an Emscripten bug, not a PHP bug, but I found it much easier to in PHP. An ideal solution would upstream Emscripten. Source: me.
Dockerfile performs string-replace ing of:
- ncurses Makefile – this could be merged into the provided . file and perhaps proposed for the upstream ncurses project. Source: same as above.
- PHP – Adjusts build nuances like enforcing posix readdir_r support on older PHP versions or manually disabling unitstd.h. A few of these changes could be brought into upstream PHP 8, but most target the EOL-ed PHP versions. Source: me.
- Built Emscripten's php5.6.js, php-8.0.js, etc. – these bugfixes and features are documented inline and I'd love to have every single one of them merged into upstream Emscripten repo as they're useful and will be hard to maintain separately. Source: me.
zlib – Playground uses a 10 years old Emscripten-compatible port of zlib because my initial attempt to compile the upstream version as WASM failed. I'm sure this old port can be removed, especially since Emscripten comes with -s USE_ZLIB=1 switch these days, but I didn't get there yet. PRs welcome!

adamziel · 2023-01-30T12:15:20Z

Let's merge this one and keep iterating in subsequent PRs

### Description Adds support for CLI SAPI and networking in node.js: ``` > npm run build > node ./build-cli/php-cli.js -r 'echo "Hello from PHP !";' Hello from PHP ! > node ./build-cli/php-cli.js -r 'echo substr(file_get_contents("https://wordpress.org"), 0, 16);' <!DOCTYPE html> > node ./build-cli/php-cli.js -r 'echo phpversion();' 8.2.0-dev > PHP=5.6 node ./build-cli/php-cli.js -r 'echo phpversion();' 5.6.40 ``` ### Highlights: * Networking is supported (including MySQL and HTTPS) * Switching PHP versions is supported. * [Most WordPress PHPUnit tests pass](WordPress/wordpress-playground#111). The failures are caused by missing extensions and a few misconfigured settings * PHP Interactive mode is supported but [the arrow keys don't work](WordPress/wordpress-playground#118) * `wp-cli` works ### In broad strokes: * CLI SAPI is compiled with libedit (readline replacement) and ncurses. * Network calls are asynchronous. Emscripten's Asyncify enables calling asynchronous code from synchronous code. TCP sockets are shimmed with a WebSocket connection to a built-in proxy server running on localhost. It supports data transfer, arbitrary connection targets, and setting a few TCP socket options. * PHP's OpenSSL uses the same CA certs as Node.js * PHP 5.6 is ed to work with OpenSSL 1.1.0 and many other small es are introduced. For more details, see [es overview](WordPress/wordpress-playground#119 (comment)), Dockerfile, and `phpwasm-emscripten-library.js` ### Future work: * PHP Interactive server isn't supported yet. Adding support is a matter of making the incoming connection polling non-blocking using Asyncify. * Use a more recent OpenSSL version * [Better support for CLI interactive mode](WordPress/wordpress-playground#118)

## Motivation for the change, related issues We want to be using the latest Playground updates on WordPress.com. Let's allow public access to our private Playground instance. ## Implementation details This PR comments out the portions of our deployment script that enforce privacy. Those portions are left in place for now to document this decision. ## Testing Instructions (or ideally a Blueprint) - CI - Deploy after merging.

adamziel added 30 commits January 14, 2023 20:36

Explore a generic php-cli npm package that enables running PHP scripts

45c5c54

Skip over the shebang line in CLI scripts

dacb78a

Configurable CLI command

d431850

Compile with the actual CLI SAPI

d09743d

Support PHP-CLI interactive mode

0e5e96e

Accept arbitrary ENV variables

48dab7c

Last steps towards a working CLI build

cd7d013

Support arbitrary number of CLI args via a linkedlist

499edd7

Cleanup PHP build dockerfile

4cb34d9

Disable unsupported PHP features via initial php.ini values

1c16b4d

Remove a typo from Docekrfile

ce92698

Remove pthreads artifats

9282cf4

Make php_wasm.c compatible with php8

331f199

Fix up dev artifacts

ca16273

Reliable CLI build for PHP 5.6-8.2

5c7b792

Restrict Node.js specific changes to node runtime

c7fb1e9

Restore the init() call when dising web requests

88b7e3e

Remove dev artifact

00b7397

Cleanup unused imports

c6fdba4

Add a build-cli directory for the php-cli builds

47b9c29

Ignore build-cli

f115faf

Add php-cli build logic

118d041

Minify PHP CLI build

1ac2593

Add a php.ini script and bump the version

86a5766

Adjust the build script to PHP 8.2 requirements

7c7b31d

Support popen()

8ef797b

Add pre-build node binaries

594665f

Enable exec functions in php.ini and bump the version

17ac5a1

Fix typo in esbuild.js

24f543c

Add the missing php-cli script

901e009

adamziel mentioned this pull request Jan 27, 2023
Generic php-cli.js script to replace the PHP terminal command #114
Closed

Updated WASM files

20dda3c

adamziel changed the title ~~Networking support in CLI~~ CLI and networking support Jan 27, 2023

Remove TERMINFO override

4949e87

adamziel marked this pull request as ready for review January 27, 2023 16:13

adamziel mentioned this pull request Jan 30, 2023
Propose PRs for the projects that are now ed ad-hoc #91
Open

adamziel added 8 commits January 30, 2023 12:15

Exit the CLI node.js process once the PHP runtime exits

dc54afa

Allowlist a missing PHP 7.0 Asyncify function

ebd9479

Remove the commented php.ini options

ec39ba4

Exit the php-cli process on error

37da784

Rebuild PHP 7.0 and 7.1

1d21d5b

Add the missing newline at the end of build-cli/package.json

d4418fc

Bump the cli package version

100c5f3

Add ws as a php-cli dependency

a626d1a

adamziel merged commit 9c7d797 into trunk Jan 30, 2023

adamziel deleted the explore/networking branch January 30, 2023 12:15

adamziel mentioned this pull request Jan 30, 2023
Support MySQL in CLI #116
Closed

artemiomorales mentioned this pull request Jan 31, 2023
Add import and export logic #110
Merged

gregvolny mentioned this pull request Jan 31, 2023
Support for external database networking r-wasm/webr#129
Open

adamziel mentioned this pull request Feb 2, 2023
Network access in the browser #85
Open

This was referencedJun 2, 2023
Gutenberg Block WordPress/playground-tools#57
Open
Feature Gallery #128
Closed

adamziel mentioned this pull request Apr 27, 2023
Tracking: Redesign Documentation #217
Closed
56 tasks

eliot-akira mentioned this pull request May 12, 2023
PHP WASM: Use Emscripten's zlib and remove bundled copy #335
Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLI and networking support #119

CLI and networking support #119

adamziel commented Jan 25, 2023•
edited
Loading

dmsnell commented Jan 27, 2023

eliot-akira commented Jan 28, 2023

adamziel commented Jan 30, 2023•
edited
Loading

adamziel commented Jan 30, 2023

CLI and networking support #119

CLI and networking support #119

Conversation

adamziel commented Jan 25, 2023•edited Loading

Description

Highlights:

In broad strokes:

Future work:

dmsnell commented Jan 27, 2023

eliot-akira commented Jan 28, 2023

adamziel commented Jan 30, 2023•edited Loading

adamziel commented Jan 30, 2023

adamziel commented Jan 25, 2023•
edited
Loading

adamziel commented Jan 30, 2023•
edited
Loading