Making firmware environment setup more friendly

Is your feature request related to a problem? Please describe.

The setup steps for Moddable SDK are still tricky for beginners. So we should help them to build the first Stack-chan application.

Describe the solution you'd like

Several solutions are considerable.

• Adding a description to README. Showing this situation makes surprises most negligible.
• Adopting environment setup tool like xs-dev. Building on the top of the docker container is more than tricky, especially for M1 Mac/Win(WSL).

@hipsterbrown has done great work in creating xs-dev:

• It automates the set-up and update of the Moddable SDK in the same way as the manual instructions.
• It starts from npm / node which makes it very easy for web developers to get started with.
• It is implemented in JavaScript (well, TypeScript 😉) which makes it easy for Moddable SDK developers to enhance.
• The Moddable team is actively supporting xs-dev development.

Beyond Moddable SDK set-up and updating, xs-dev has features in-progress to make it easy to create and manage new projects.

Let me know if there's something should be added or fixed to make this more accessible for use in this project. 😃 I would love to test out the setup if I can find a M5 Stack available

I tried introducing xs-dev on the latest dev/v1.0 branch. I am using xs-dev to set up and diagnose the Moddable environment in the project's npm script. It works like a charm in my environment (Ubuntu 20.04)! I did not use 'xs-dev run' because I did not know how to only run a specific phase of the build. I would be grateful if you could point out any issues in the project's settings.

@meganetaaan, thank you for trying out xs-dev. Great to hear that it is mostly working for you.

...I did not know how to only run a specific phase of the build...

You are referring to the build targets capability of mcconfig, such as -t xsbug? I think that should be straightforward to support, but we should hear from @HipsterBrown about the best way to do that.

I tried to install environment under Docker(24.0.2) on Ubuntu 20.04

I faced this error with npm run setup -- --device=esp32 command

> [email protected] setup
> xs-dev setup --device=esp32

ℹ Ensuring esp32 install directory
✔ Cloning esp-idf repo
Reading package lists... Donees
Building dependency tree       
Reading state information... Done
bison is already the newest version (2:3.5.1+dfsg-1).
ccache is already the newest version (3.7.7-1).
cmake is already the newest version (3.16.3-1ubuntu1.20.04.1).
dfu-util is already the newest version (0.9-1).
flex is already the newest version (2.6.4-6.2).
git is already the newest version (1:2.25.1-1ubuntu3.10).
gperf is already the newest version (3.1-1build1).
libffi-dev is already the newest version (3.3-4).
libssl-dev is already the newest version (1.1.1f-1ubuntu2.19).
ninja-build is already the newest version (1.10.0-1build1).
python-is-python3 is already the newest version (3.8.2-4).
python-setuptools is already the newest version (44.0.0-2ubuntu0.1).
python3-pip is already the newest version (20.0.2-5ubuntu1.8).
python3-serial is already the newest version (3.4-5.1).
wget is already the newest version (1.20.3-1ubuntu2).
⠙ Installing build dependencies0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
✔ Installing build dependencies
⠋ Installing esp-idf toolingDetecting the Python interpreter
Checking "python" ...
Python 3.8.10
"python" has been detected
Installing ESP-IDF tools
⠙ Installing esp-idf toolingCurrent system platform: linux-amd64
⠹ Installing esp-idf tooling/workspace/node_modules/gluegun/build/index.js:15
    throw up;
    ^

Error: Command failed with exit code 1: ./install.sh
Traceback (most recent call last):
  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1860, in <module>
    main(sys.argv[1:])
  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1856, in main
    action_func(args)
  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1423, in action_install
    targets = clean_targets(args.targets)
  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1085, in clean_targets
    export_targets_to_idf_env_json(targets_from_tools_json)
  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1053, in export_targets_to_idf_env_json
    targets = list(set(targets + get_user_defined_targets()))
  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1099, in get_user_defined_targets
    if env == idf_env_json['idfSelectedId']:
KeyError: 'idfSelectedId'
    at makeError (/workspace/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/workspace/node_modules/execa/index.js:118:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  shortMessage: 'Command failed with exit code 1: ./install.sh',
  command: './install.sh',
  escapedCommand: '"./install.sh"',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: undefined,
  stderr: 'Traceback (most recent call last):\n' +
    '  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1860, in <module>\n' +
    '    main(sys.argv[1:])\n' +
    '  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1856, in main\n' +
    '    action_func(args)\n' +
    '  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1423, in action_install\n' +
    '    targets = clean_targets(args.targets)\n' +
    '  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1085, in clean_targets\n' +
    '    export_targets_to_idf_env_json(targets_from_tools_json)\n' +
    '  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1053, in export_targets_to_idf_env_json\n' +
    '    targets = list(set(targets + get_user_defined_targets()))\n' +
    '  File "/root/.local/share/esp32/esp-idf/tools/idf_tools.py", line 1099, in get_user_defined_targets\n' +
    "    if env == idf_env_json['idfSelectedId']:\n" +
    "KeyError: 'idfSelectedId'",
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}

Node.js v18.16.1

Could you please tell me how to fix this error?

@HipsterBrown - any advice?

I'll need to try running through this myself. The error output from the esp-idf isn't providing a lot of context.

I should probably start setting up some E2E testing for the setup scenarios to make sure they continue to work across OS and device options.

I believe I discovered the issue.

The Docker image used by the firmware/docker/Dockerfile contains a version (v4.4.3) of the idf-env.json at the IDF_TOOLS_PATH that the esp-idf install script uses to determine the installed environment devices. The version of the esp-idf pulled in by xs-dev is different (v4.4.2), so the install script throws an error when it doesn't find an expected key. https://github.com/espressif/esp-idf/issues/9837

This can be fixed in xs-dev by updating the esp-idf tagged version to v4.4.3, which will then require the stack-chan firmware to update its xs-dev dependency version to that released fix version (probably v0.25.13).

While working on this, I noticed the published Docker image from the docker-moddable-esp32 project sets up the Moddable tooling itself from the public branch at the time of building the image: https://github.com/meganetaaan/docker-moddable-esp32/blob/master/Dockerfile.head#L32-L36 This could become tricky to coordinate with xs-dev, which attempts to use the latest tagged releases of Moddable. So I would probably document that npm run setup is not required (I believe) or recommended when using the Docker environment.

Official xs-dev Docker images is on the list of proposed integrations to make this a more seamless experience.

@HipsterBrown thank you for comment. How I check esp-idf version under docker envrionment... I user fix version(0.25.13) attached Dockerfile Dockerfile.txt