Global refactoring (#617)

* Rename `bench-build` target to `bench` in bake file

* Update bake file and break everything

* Rename docker-compose.yml to compose.yml to avoid conflicting on `docker buildx bake`

* Fix groups in bake file

* Update frappe-worker

* Update frappe-nginx, erpnext-nginx

* Remove old erpnext images

* Update frappe-socketio

* Fix develop frappe-nginx build on linux/arm64

* Update dockerignore

* Update gitignore

* Update gitignore

* Update .env files

* Update installation (overrides)

* Update tests

* Fix image names

* Update compose

* Update get-latest-tags

* Update CI

* Setup and remove .env on tests

* Add build bench workflow

* Add triggers to main workflow

* Add release helm job

* Use reusable workflows

* Rollback

* Print configuration before running tests

* Show tests/.env

* Revert "Show tests/.env"

This reverts commit 4bc3bdebaf.

* Fix ci image versions

* Remove `frappe-` prefix in build directories

* Move requirements-dev.txt

* Fix image name in CI

* Update gitignore

* Update pre-commit config

* Drop `version:` in compose files

* Add push-backup

* Fix postgres CI test

* Change .yml to .yaml in compose file to follow compose-spec

* Remove prettierignore

* Fix dockerignore

* Change .yml to .yaml in compose file to follow compose-spec

* Don't depend on boto3 while testing (do it in backend)

* Update erpnext example version

* Don't fail ping on URLError

* Move assets volume to main compose file

* Fix type annotations for v12

* Fix postgres ci override in tests

* Fix spaces in socketio

* Reorder stages in nginx image, improve perfomance

* Remove unused todo

* Optimize worker build

* Install Node in worker image

* Add 502 error page

* Remove unused quiet-pull in tests

* Add configurator service to dynamically set common config

* Remove unused compose.ci-postgres.yml

* Use Python for configurator service: faster and more robust

* Add TODO.md

* Use python script to get latest tags in CI

* Clean up nginx dockerfile

* Remove VOLUME declaration

https://stackoverflow.com/a/55052682

* Add custom app example

* Remove pwd for now

* Remove pwd for now

* Use jq for parsing config in healthcheck

* Take advantage of yaml lang: add defaults in compose file. Also require env vars

* Fix CI

* Use resusable workflow

* Update

* Move release_helm job to main.yml

* Rename docker-build to docker-build-push

* Rename main to build_stable

* Rename bench targets

* Remove quotes from docker-build-push inputs

* Update build develop

* Remove HELM_DEPLOY_KEY secret from docker-build-push

* Add job names

* Remove build_bench workflow

* Update version input description in docker-build-push

* Print .env in tests, if version is develop, change to latest (for tag)

* Fix env setup

* Uncomment tests

* Parse and set short tags from git tag in bake file

* Move devcontainer settings to devcontainer.json

* Add db command notice

* Fix CI?

* Fix inconsistencies in development readme

* Remove pwd for now

* Remove custom apps for production instruction

* Update todos

* Add docs for images and compose files

* Add variables docs and allow custom frappe site name header

* Add notice about internal environment variables

* Update site-operations docs

* Update todos

* Add Overrides header in images-and-compose-files

* Update todos

* Remove extra docs

* Don't log requests in worker image (nginx already does that)

* Remove default value of FRAPPE_SITE_NAME_HEADER in example.env

* Use file that consistent in v12, v13 and develop to check /assets

* Fix paths in CI

* Update todos

* Remove TODO.md

* Update tests/_check_backup_files.py

Co-authored-by: Revant Nandgaonkar <revant.one@gmail.com>

* Change variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY to S3_ACCESS_KEY, S3_SECRET_KEY in tests

* Fix S3 test

* Use `nginxinc/nginx-unprivileged` instead of `nginx` image

Also use Ngnix 1.20 instead of unstable 1.21

* Fix https override

* Update Dockerfile

* Mount assets to backend service in read only mode

* Touch .build (#307), use scripts from nginx image to generate config and touch .build

* Update example env after building stable images

* Touch `.build` on develop image (untill https://github.com/frappe/frappe/issues/15396 is resolved)

* Add `make` to worker build deps for linux/arm64

* Fix update example.env job

* Fix .build creation on develop branch

* Move bench CI to different file

This way workflow runs only on PRs that relevant to bench build

* Fix app name in custom app example

* Update erpnext and frappe versions in example.env

* Don't install `svg-sprite` and `sass` node modules in nginx image on linux/arm64 (https://github.com/frappe/frappe/pull/15275)

* docs: README and docs

* docs: add link to site operations from docker swarm

* ci: fix tests as per changes to compose.yaml

* docs: move wiki articles to docs

* docs: fix add custom domain

* docs: fix patch code from images

* fix: do not expose port 80 for old images

* fix: custom domain labels to frontend container/service

* Add missing descriptions to envs in example.env

* Fix redis depends_on

* Fix docker compose in tests when not running on TTY

* Set -T flag in `docker compose exec` only if not tty

* Run pre-commit on docs

* Remove postgres healthcheck (it gets overriden by mariadb)

* Refactor test

* Update workflow names

* Add pip to dependabot config

* docs: backup and push (#19)

* Beautify changes by @revant (#20)

* feat: add gevent to worker image

* feat: real_ip configuration for nginx

* Return `healthcheck.sh` just for tests

Co-authored-by: Lev Vereshchagin <mail@vrslev.com>

* Make pretend bench catch unknown commands (closes #666)

* Remove debug print in push-backup

* Fix typing issues in push-backup

* Update file keys in push-backups: from abs path to <site>/<file>

* Refactor push-backup

* Move gevent installation in Frappe step

* Don't pin boto stubs requirement

* Cache pip deps on build

* Update example env versions

* Refactor check backup files

* Fix backup test

* Fix backup test

* Rename build/ dir to images/

* Rename build/ dir to images/

* Fix /build -> /images in docs

* Update example.env

* Use reusable workflow in frappe user instead of vrslev

* Fix compose`s `project` option in docs (https://github.com/frappe/frappe_docker/pull/617#issuecomment-1065178792)

* Add note about project option in site-operations doc

* Update example env

* Rename build arg `USERNAME` to `REGISTRY_USER`

* Allow https proxy to access Docker socket

* Revert "Use reusable workflow in frappe user instead of vrslev"

This reverts commit 6062500d0d.

* Revert "Revert "Use reusable workflow in frappe user instead of vrslev""

This reverts commit 4680d18ff8.

Co-authored-by: Revant Nandgaonkar <revant.one@gmail.com>

custom_app/README.md

@@ -0,0 +1,38 @@
+This is basic configuration for building images and testing custom apps that use Frappe.
+
+You can see that there's four files in this folder:
+
+- `backend.Dockerfile`,
+- `frontend.Dockerfile`,
+- `docker-bake.hcl`,
+- `compose.override.yaml`.
+
+Python code will `backend.Dockerfile`. JS and CSS (and other fancy frontend stuff) files will be built in `frontend.Dockerfile` if required and served from there.
+
+`docker-bake.hcl` is reference file for cool new [Buildx Bake](https://github.com/docker/buildx/blob/master/docs/reference/buildx_bake.md). It helps to build images without having to remember all build arguments.
+
+`compose.override.yaml` is [Compose](https://docs.docker.com/compose/compose-file/) override that replaces images from [main compose file](https://github.com/frappe/frappe_docker/blob/main/compose.yaml) so it would use your own images.
+
+To get started, install Docker and [Buildx](https://github.com/docker/buildx#installing). Then copy all content of this folder (except this README) to your app's root directory. Also copy `compose.yaml` in the root of this repository.
+
+Before the next step—to build images—replace "custom_app" with your app's name in `docker-bake.hcl`. After that, let's try to build:
+
+```bash
+FRAPPE_VERSION=<Frappe version you need> docker buildx bake
+```
+
+If something goes wrong feel free to leave an issue.
+
+To test if site works, setup `.env` file (check [example](<(https://github.com/frappe/frappe_docker/blob/main/example.env)>)) and run:
+
+```bash
+docker-compose up -d
+docker-compose exec backend \
+  bench new-site 127.0.0.1 \
+    --mariadb-root-password 123 \
+    --admin-password admin \
+    --install-app <Name of your app>
+docker-compose restart backend
+```
+
+Cool! You just containerized your app!

custom_app/backend.Dockerfile

@@ -0,0 +1,8 @@
+ARG FRAPPE_VERSION
+FROM frappe/frappe-worker:${FRAPPE_VERSION}
+
+ARG APP_NAME
+COPY --chown=frappe . ../apps/${APP_NAME}
+
+RUN echo "frappe\n${APP_NAME}" >/home/frappe/frappe-bench/sites/apps.txt \
+    && ../env/bin/pip install --no-cache-dir -e ../apps/${APP_NAME}

custom_app/compose.override.yaml

@@ -0,0 +1,21 @@
+services:
+  configurator:
+    image: custom_app/worker:${VERSION}
+
+  backend:
+    image: custom_app/worker:${VERSION}
+
+  frontend:
+    image: custom_app/nginx:${VERSION}
+
+  queue-short:
+    image: custom_app/worker:${VERSION}
+
+  queue-default:
+    image: custom_app/worker:${VERSION}
+
+  queue-long:
+    image: custom_app/worker:${VERSION}
+
+  scheduler:
+    image: custom_app/worker:${VERSION}

custom_app/docker-bake.hcl

@@ -0,0 +1,25 @@
+APP_NAME="custom_app"
+
+variable "FRAPPE_VERSION" {}
+
+group "default" {
+    targets = ["backend", "frontend"]
+}
+
+target "backend" {
+    dockerfile = "backend.Dockerfile"
+    tags = ["custom_app/worker:latest"]
+    args = {
+      "FRAPPE_VERSION" = FRAPPE_VERSION
+      "APP_NAME" = APP_NAME
+    }
+}
+
+target "frontend" {
+    dockerfile = "frontend.Dockerfile"
+    tags = ["custom_app/nginx:latest"]
+    args = {
+      "FRAPPE_VERSION" = FRAPPE_VERSION
+      "APP_NAME" = APP_NAME
+    }
+}

custom_app/frontend.Dockerfile

@@ -0,0 +1,52 @@
+ARG FRAPPE_VERSION
+FROM node:14-bullseye-slim as prod_node_modules
+
+RUN apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
+    git \
+    build-essential \
+    python \
+    ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /root/frappe-bench
+RUN mkdir -p sites/assets
+
+ARG FRAPPE_VERSION
+RUN git clone --depth 1 -b ${FRAPPE_VERSION} https://github.com/frappe/frappe apps/frappe
+
+RUN yarn --cwd apps/frappe
+
+
+ARG APP_NAME
+COPY . apps/${APP_NAME}
+
+# Install production node modules
+RUN yarn --cwd apps/${APP_NAME} --prod
+
+
+
+FROM prod_node_modules as assets
+
+ARG APP_NAME
+
+# Install development node modules
+RUN yarn --cwd apps/${APP_NAME}
+
+# Build assets
+RUN echo "frappe\n${APP_NAME}" >sites/apps.txt \
+    && yarn --cwd apps/frappe production --app ${APP_NAME} \
+    && rm sites/apps.txt
+
+
+
+FROM frappe/frappe-nginx:${FRAPPE_VERSION}
+
+ARG APP_NAME
+
+# Copy all not built assets
+COPY --from=prod_node_modules /root/frappe-bench/apps/${APP_NAME}/${APP_NAME}/public /usr/share/nginx/html/assets/${APP_NAME}
+# Copy production node modules
+COPY --from=prod_node_modules /root/frappe-bench/apps/${APP_NAME}/node_modules /usr/share/nginx/html/assets/${APP_NAME}/node_modules
+# Copy built assets
+COPY --from=assets /root/frappe-bench/sites /usr/share/nginx/html