Global refactoring (#617)
* Rename `bench-build` target to `bench` in bake file * Update bake file and break everything * Rename docker-compose.yml to compose.yml to avoid conflicting on `docker buildx bake` * Fix groups in bake file * Update frappe-worker * Update frappe-nginx, erpnext-nginx * Remove old erpnext images * Update frappe-socketio * Fix develop frappe-nginx build on linux/arm64 * Update dockerignore * Update gitignore * Update gitignore * Update .env files * Update installation (overrides) * Update tests * Fix image names * Update compose * Update get-latest-tags * Update CI * Setup and remove .env on tests * Add build bench workflow * Add triggers to main workflow * Add release helm job * Use reusable workflows * Rollback * Print configuration before running tests * Show tests/.env * Revert "Show tests/.env" This reverts commit4bc3bdebaf. * Fix ci image versions * Remove `frappe-` prefix in build directories * Move requirements-dev.txt * Fix image name in CI * Update gitignore * Update pre-commit config * Drop `version:` in compose files * Add push-backup * Fix postgres CI test * Change .yml to .yaml in compose file to follow compose-spec * Remove prettierignore * Fix dockerignore * Change .yml to .yaml in compose file to follow compose-spec * Don't depend on boto3 while testing (do it in backend) * Update erpnext example version * Don't fail ping on URLError * Move assets volume to main compose file * Fix type annotations for v12 * Fix postgres ci override in tests * Fix spaces in socketio * Reorder stages in nginx image, improve perfomance * Remove unused todo * Optimize worker build * Install Node in worker image * Add 502 error page * Remove unused quiet-pull in tests * Add configurator service to dynamically set common config * Remove unused compose.ci-postgres.yml * Use Python for configurator service: faster and more robust * Add TODO.md * Use python script to get latest tags in CI * Clean up nginx dockerfile * Remove VOLUME declaration https://stackoverflow.com/a/55052682 * Add custom app example * Remove pwd for now * Remove pwd for now * Use jq for parsing config in healthcheck * Take advantage of yaml lang: add defaults in compose file. Also require env vars * Fix CI * Use resusable workflow * Update * Move release_helm job to main.yml * Rename docker-build to docker-build-push * Rename main to build_stable * Rename bench targets * Remove quotes from docker-build-push inputs * Update build develop * Remove HELM_DEPLOY_KEY secret from docker-build-push * Add job names * Remove build_bench workflow * Update version input description in docker-build-push * Print .env in tests, if version is develop, change to latest (for tag) * Fix env setup * Uncomment tests * Parse and set short tags from git tag in bake file * Move devcontainer settings to devcontainer.json * Add db command notice * Fix CI? * Fix inconsistencies in development readme * Remove pwd for now * Remove custom apps for production instruction * Update todos * Add docs for images and compose files * Add variables docs and allow custom frappe site name header * Add notice about internal environment variables * Update site-operations docs * Update todos * Add Overrides header in images-and-compose-files * Update todos * Remove extra docs * Don't log requests in worker image (nginx already does that) * Remove default value of FRAPPE_SITE_NAME_HEADER in example.env * Use file that consistent in v12, v13 and develop to check /assets * Fix paths in CI * Update todos * Remove TODO.md * Update tests/_check_backup_files.py Co-authored-by: Revant Nandgaonkar <revant.one@gmail.com> * Change variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY to S3_ACCESS_KEY, S3_SECRET_KEY in tests * Fix S3 test * Use `nginxinc/nginx-unprivileged` instead of `nginx` image Also use Ngnix 1.20 instead of unstable 1.21 * Fix https override * Update Dockerfile * Mount assets to backend service in read only mode * Touch .build (#307), use scripts from nginx image to generate config and touch .build * Update example env after building stable images * Touch `.build` on develop image (untill https://github.com/frappe/frappe/issues/15396 is resolved) * Add `make` to worker build deps for linux/arm64 * Fix update example.env job * Fix .build creation on develop branch * Move bench CI to different file This way workflow runs only on PRs that relevant to bench build * Fix app name in custom app example * Update erpnext and frappe versions in example.env * Don't install `svg-sprite` and `sass` node modules in nginx image on linux/arm64 (https://github.com/frappe/frappe/pull/15275) * docs: README and docs * docs: add link to site operations from docker swarm * ci: fix tests as per changes to compose.yaml * docs: move wiki articles to docs * docs: fix add custom domain * docs: fix patch code from images * fix: do not expose port 80 for old images * fix: custom domain labels to frontend container/service * Add missing descriptions to envs in example.env * Fix redis depends_on * Fix docker compose in tests when not running on TTY * Set -T flag in `docker compose exec` only if not tty * Run pre-commit on docs * Remove postgres healthcheck (it gets overriden by mariadb) * Refactor test * Update workflow names * Add pip to dependabot config * docs: backup and push (#19) * Beautify changes by @revant (#20) * feat: add gevent to worker image * feat: real_ip configuration for nginx * Return `healthcheck.sh` just for tests Co-authored-by: Lev Vereshchagin <mail@vrslev.com> * Make pretend bench catch unknown commands (closes #666) * Remove debug print in push-backup * Fix typing issues in push-backup * Update file keys in push-backups: from abs path to <site>/<file> * Refactor push-backup * Move gevent installation in Frappe step * Don't pin boto stubs requirement * Cache pip deps on build * Update example env versions * Refactor check backup files * Fix backup test * Fix backup test * Rename build/ dir to images/ * Rename build/ dir to images/ * Fix /build -> /images in docs * Update example.env * Use reusable workflow in frappe user instead of vrslev * Fix compose`s `project` option in docs (https://github.com/frappe/frappe_docker/pull/617#issuecomment-1065178792) * Add note about project option in site-operations doc * Update example env * Rename build arg `USERNAME` to `REGISTRY_USER` * Allow https proxy to access Docker socket * Revert "Use reusable workflow in frappe user instead of vrslev" This reverts commit6062500d0d. * Revert "Revert "Use reusable workflow in frappe user instead of vrslev"" This reverts commit4680d18ff8. Co-authored-by: Revant Nandgaonkar <revant.one@gmail.com>
This commit is contained in:
Lev
123
images/worker/Dockerfile
Normal file
123
images/worker/Dockerfile
Normal file
@@ -0,0 +1,123 @@
|
||||
# syntax=docker/dockerfile:1.3
|
||||
|
||||
ARG PYTHON_VERSION
|
||||
FROM python:${PYTHON_VERSION}-slim-bullseye as base
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install --no-install-recommends -y \
|
||||
# Postgres
|
||||
libpq-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN useradd -ms /bin/bash frappe
|
||||
USER frappe
|
||||
RUN mkdir -p /home/frappe/frappe-bench/apps /home/frappe/frappe-bench/logs /home/frappe/frappe-bench/sites
|
||||
WORKDIR /home/frappe/frappe-bench
|
||||
|
||||
RUN --mount=type=cache,target=/home/frappe/.cache/pip \
|
||||
pip install -U pip wheel \
|
||||
&& python -m venv env \
|
||||
&& env/bin/pip install -U pip wheel
|
||||
USER root
|
||||
|
||||
|
||||
|
||||
FROM base as build_deps
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install --no-install-recommends -y \
|
||||
# Install git here because it is not required in production
|
||||
git \
|
||||
# gcc and g++ are required for building different packages across different versions
|
||||
# of Frappe and ERPNext and also on different platforms (for example, linux/arm64).
|
||||
# It is safe to install build deps even if they are not required
|
||||
# because they won't be included in final images.
|
||||
gcc \
|
||||
g++ \
|
||||
# Make is required to build wheels of ERPNext deps in develop branch for linux/arm64
|
||||
make \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
USER frappe
|
||||
|
||||
|
||||
|
||||
FROM build_deps as frappe_builder
|
||||
|
||||
ARG FRAPPE_VERSION
|
||||
RUN --mount=type=cache,target=/home/frappe/.cache/pip \
|
||||
git clone --depth 1 -b ${FRAPPE_VERSION} https://github.com/frappe/frappe apps/frappe \
|
||||
&& env/bin/pip install -e apps/frappe \
|
||||
&& env/bin/pip install -U gevent \
|
||||
&& rm -r apps/frappe/.git
|
||||
|
||||
|
||||
|
||||
# We split ERPNext wheels build in separate stage to achieve concurrency with Frappe build
|
||||
FROM build_deps as erpnext_wheels
|
||||
|
||||
ARG ERPNEXT_VERSION
|
||||
RUN git clone --depth 1 -b ${ERPNEXT_VERSION} https://github.com/frappe/erpnext apps/erpnext \
|
||||
&& rm -r apps/erpnext/.git
|
||||
|
||||
RUN --mount=type=cache,target=/home/frappe/.cache/pip \
|
||||
pip wheel --wheel-dir /home/frappe/erpnext-wheels -r apps/erpnext/requirements.txt
|
||||
|
||||
|
||||
|
||||
FROM frappe_builder as erpnext_builder
|
||||
|
||||
COPY --from=erpnext_wheels --chown=frappe /home/frappe/frappe-bench/apps/erpnext /home/frappe/frappe-bench/apps/erpnext
|
||||
RUN --mount=type=bind,target=/home/frappe/erpnext-wheels,source=/home/frappe/erpnext-wheels,from=erpnext_wheels \
|
||||
--mount=type=cache,target=/home/frappe/.cache/pip \
|
||||
--mount=type=cache,target=/home/frappe/.cache/pip,source=/home/frappe/.cache/pip,from=erpnext_wheels \
|
||||
env/bin/pip install --find-links=/home/frappe/erpnext-wheels -e apps/erpnext
|
||||
|
||||
|
||||
|
||||
FROM base as configured_base
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install --no-install-recommends -y curl \
|
||||
&& curl -sL https://deb.nodesource.com/setup_14.x | bash - \
|
||||
&& apt-get purge -y --auto-remove curl \
|
||||
&& apt-get update \
|
||||
&& apt-get install --no-install-recommends -y \
|
||||
# MariaDB
|
||||
mariadb-client \
|
||||
# Postgres
|
||||
postgresql-client \
|
||||
# wkhtmltopdf
|
||||
xvfb \
|
||||
libfontconfig \
|
||||
wkhtmltopdf \
|
||||
# For healthcheck
|
||||
wait-for-it \
|
||||
jq \
|
||||
# other
|
||||
nodejs \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
USER frappe
|
||||
|
||||
COPY pretend-bench.sh /usr/local/bin/bench
|
||||
COPY push_backup.py /usr/local/bin/push-backup
|
||||
COPY configure.py patched_bench_helper.py /usr/local/bin/
|
||||
COPY gevent_patch.py /opt/patches/
|
||||
|
||||
WORKDIR /home/frappe/frappe-bench/sites
|
||||
|
||||
CMD [ "/home/frappe/frappe-bench/env/bin/gunicorn", "-b", "0.0.0.0:8000", "frappe.app:application" ]
|
||||
|
||||
|
||||
FROM configured_base as frappe
|
||||
|
||||
RUN echo "frappe" >/home/frappe/frappe-bench/sites/apps.txt
|
||||
COPY --from=frappe_builder /home/frappe/frappe-bench/apps/frappe /home/frappe/frappe-bench/apps/frappe
|
||||
COPY --from=frappe_builder /home/frappe/frappe-bench/env /home/frappe/frappe-bench/env
|
||||
|
||||
|
||||
FROM configured_base as erpnext
|
||||
|
||||
RUN echo "frappe\nerpnext" >/home/frappe/frappe-bench/sites/apps.txt
|
||||
COPY --from=frappe_builder /home/frappe/frappe-bench/apps/frappe /home/frappe/frappe-bench/apps/frappe
|
||||
COPY --from=erpnext_builder /home/frappe/frappe-bench/apps/erpnext /home/frappe/frappe-bench/apps/erpnext
|
||||
COPY --from=erpnext_builder /home/frappe/frappe-bench/env /home/frappe/frappe-bench/env
|
||||
55
images/worker/configure.py
Executable file
55
images/worker/configure.py
Executable file
@@ -0,0 +1,55 @@
|
||||
#!/usr/local/bin/python
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Type, TypeVar
|
||||
|
||||
|
||||
def update_config(**values: Any):
|
||||
fname = "common_site_config.json"
|
||||
if not os.path.exists(fname):
|
||||
with open(fname, "a") as f:
|
||||
json.dump({}, f)
|
||||
|
||||
with open(fname, "r+") as f:
|
||||
config: dict[str, Any] = json.load(f)
|
||||
config.update(values)
|
||||
f.seek(0)
|
||||
f.truncate()
|
||||
json.dump(config, f)
|
||||
|
||||
|
||||
_T = TypeVar("_T")
|
||||
|
||||
|
||||
def env(name: str, type_: Type[_T] = str) -> _T:
|
||||
value = os.getenv(name)
|
||||
if not value:
|
||||
raise RuntimeError(f'Required environment variable "{name}" not set')
|
||||
try:
|
||||
value = type_(value)
|
||||
except Exception:
|
||||
raise RuntimeError(
|
||||
f'Cannot convert environment variable "{name}" to type "{type_}"'
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def generate_redis_url(url: str):
|
||||
return f"redis://{url}"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
update_config(
|
||||
db_host=env("DB_HOST"),
|
||||
db_port=env("DB_PORT", int),
|
||||
redis_cache=generate_redis_url(env("REDIS_CACHE")),
|
||||
redis_queue=generate_redis_url(env("REDIS_QUEUE")),
|
||||
redis_socketio=generate_redis_url(env("REDIS_SOCKETIO")),
|
||||
socketio_port=env("SOCKETIO_PORT", int),
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
3
images/worker/gevent_patch.py
Normal file
3
images/worker/gevent_patch.py
Normal file
@@ -0,0 +1,3 @@
|
||||
import gevent.monkey
|
||||
|
||||
gevent.monkey.patch_all()
|
||||
46
images/worker/patched_bench_helper.py
Normal file
46
images/worker/patched_bench_helper.py
Normal file
@@ -0,0 +1,46 @@
|
||||
import click
|
||||
import click.exceptions
|
||||
import frappe.app
|
||||
import frappe.database.db_manager
|
||||
import frappe.utils.bench_helper
|
||||
|
||||
|
||||
def patch_database_creator():
|
||||
"""
|
||||
We need to interrupt Frappe site database creation to monkeypatch
|
||||
functions that resolve host for user that owns site database.
|
||||
In frappe_docker this was implemented in "new" command:
|
||||
https://github.com/frappe/frappe_docker/blob/c808ad1767feaf793a2d14541ac0f4d9cbab45b3/build/frappe-worker/commands/new.py#L87
|
||||
"""
|
||||
|
||||
frappe.database.db_manager.DbManager.get_current_host = lambda self: "%"
|
||||
|
||||
|
||||
def patch_click_usage_error():
|
||||
bits: tuple[str, ...] = (
|
||||
click.style(
|
||||
"Only Frappe framework bench commands are available in container setup.",
|
||||
fg="yellow",
|
||||
bold=True,
|
||||
),
|
||||
"https://frappeframework.com/docs/v13/user/en/bench/frappe-commands",
|
||||
)
|
||||
notice = "\n".join(bits)
|
||||
|
||||
def format_message(self: click.exceptions.UsageError):
|
||||
if "No such command" in self.message:
|
||||
return f"{notice}\n\n{self.message}"
|
||||
return self.message
|
||||
|
||||
click.exceptions.UsageError.format_message = format_message
|
||||
|
||||
|
||||
def main() -> int:
|
||||
patch_database_creator()
|
||||
patch_click_usage_error()
|
||||
frappe.utils.bench_helper.main()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
5
images/worker/pretend-bench.sh
Executable file
5
images/worker/pretend-bench.sh
Executable file
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# shellcheck disable=SC2068
|
||||
~/frappe-bench/env/bin/python /usr/local/bin/patched_bench_helper.py frappe $@
|
||||
105
images/worker/push_backup.py
Executable file
105
images/worker/push_backup.py
Executable file
@@ -0,0 +1,105 @@
|
||||
#!/home/frappe/frappe-bench/env/bin/python
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, List, cast
|
||||
|
||||
import boto3
|
||||
import frappe
|
||||
from frappe.utils.backups import BackupGenerator
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from mypy_boto3_s3.service_resource import _Bucket
|
||||
|
||||
|
||||
class Arguments(argparse.Namespace):
|
||||
site: str
|
||||
bucket: str
|
||||
region_name: str
|
||||
endpoint_url: str
|
||||
aws_access_key_id: str
|
||||
aws_secret_access_key: str
|
||||
|
||||
|
||||
def _get_files_from_previous_backup(site_name: str) -> list[Path]:
|
||||
frappe.connect(site_name)
|
||||
|
||||
conf = cast(Any, frappe.conf)
|
||||
backup_generator = BackupGenerator(
|
||||
db_name=conf.db_name,
|
||||
user=conf.db_name,
|
||||
password=conf.db_password,
|
||||
db_host=frappe.db.host,
|
||||
db_port=frappe.db.port,
|
||||
db_type=conf.db_type,
|
||||
)
|
||||
recent_backup_files = backup_generator.get_recent_backup(24)
|
||||
|
||||
frappe.destroy()
|
||||
return [Path(f) for f in recent_backup_files if f]
|
||||
|
||||
|
||||
def get_files_from_previous_backup(site_name: str) -> list[Path]:
|
||||
files = _get_files_from_previous_backup(site_name)
|
||||
if not files:
|
||||
print("No backup found that was taken <24 hours ago.")
|
||||
return files
|
||||
|
||||
|
||||
def get_bucket(args: Arguments) -> "_Bucket":
|
||||
return boto3.resource(
|
||||
service_name="s3",
|
||||
endpoint_url=args.endpoint_url,
|
||||
region_name=args.region_name,
|
||||
aws_access_key_id=args.aws_access_key_id,
|
||||
aws_secret_access_key=args.aws_secret_access_key,
|
||||
).Bucket(args.bucket)
|
||||
|
||||
|
||||
def upload_file(path: Path, site_name: str, bucket: "_Bucket") -> None:
|
||||
filename = str(path.absolute())
|
||||
key = str(Path(site_name) / path.name)
|
||||
print(f"Uploading {key}")
|
||||
bucket.upload_file(Filename=filename, Key=key)
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def push_backup(args: Arguments) -> None:
|
||||
"""Get latest backup files using Frappe utils, push them to S3 and remove local copy"""
|
||||
|
||||
files = get_files_from_previous_backup(args.site)
|
||||
bucket = get_bucket(args)
|
||||
|
||||
for path in files:
|
||||
upload_file(path=path, site_name=args.site, bucket=bucket)
|
||||
|
||||
print("Done!")
|
||||
|
||||
|
||||
def parse_args(args: List[str]) -> Arguments:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--site", required=True)
|
||||
parser.add_argument("--bucket", required=True)
|
||||
parser.add_argument("--region-name", required=True)
|
||||
parser.add_argument("--endpoint-url", required=True)
|
||||
# Looking for default AWS credentials variables
|
||||
parser.add_argument(
|
||||
"--aws-access-key-id", required=True, default=os.getenv("AWS_ACCESS_KEY_ID")
|
||||
)
|
||||
parser.add_argument(
|
||||
"--aws-secret-access-key",
|
||||
required=True,
|
||||
default=os.getenv("AWS_SECRET_ACCESS_KEY"),
|
||||
)
|
||||
return parser.parse_args(args, namespace=Arguments())
|
||||
|
||||
|
||||
def main(args: List[str]) -> int:
|
||||
push_backup(parse_args(args))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
Reference in New Issue
Block a user