Global refactoring (#617)
* Rename `bench-build` target to `bench` in bake file * Update bake file and break everything * Rename docker-compose.yml to compose.yml to avoid conflicting on `docker buildx bake` * Fix groups in bake file * Update frappe-worker * Update frappe-nginx, erpnext-nginx * Remove old erpnext images * Update frappe-socketio * Fix develop frappe-nginx build on linux/arm64 * Update dockerignore * Update gitignore * Update gitignore * Update .env files * Update installation (overrides) * Update tests * Fix image names * Update compose * Update get-latest-tags * Update CI * Setup and remove .env on tests * Add build bench workflow * Add triggers to main workflow * Add release helm job * Use reusable workflows * Rollback * Print configuration before running tests * Show tests/.env * Revert "Show tests/.env" This reverts commit4bc3bdebaf. * Fix ci image versions * Remove `frappe-` prefix in build directories * Move requirements-dev.txt * Fix image name in CI * Update gitignore * Update pre-commit config * Drop `version:` in compose files * Add push-backup * Fix postgres CI test * Change .yml to .yaml in compose file to follow compose-spec * Remove prettierignore * Fix dockerignore * Change .yml to .yaml in compose file to follow compose-spec * Don't depend on boto3 while testing (do it in backend) * Update erpnext example version * Don't fail ping on URLError * Move assets volume to main compose file * Fix type annotations for v12 * Fix postgres ci override in tests * Fix spaces in socketio * Reorder stages in nginx image, improve perfomance * Remove unused todo * Optimize worker build * Install Node in worker image * Add 502 error page * Remove unused quiet-pull in tests * Add configurator service to dynamically set common config * Remove unused compose.ci-postgres.yml * Use Python for configurator service: faster and more robust * Add TODO.md * Use python script to get latest tags in CI * Clean up nginx dockerfile * Remove VOLUME declaration https://stackoverflow.com/a/55052682 * Add custom app example * Remove pwd for now * Remove pwd for now * Use jq for parsing config in healthcheck * Take advantage of yaml lang: add defaults in compose file. Also require env vars * Fix CI * Use resusable workflow * Update * Move release_helm job to main.yml * Rename docker-build to docker-build-push * Rename main to build_stable * Rename bench targets * Remove quotes from docker-build-push inputs * Update build develop * Remove HELM_DEPLOY_KEY secret from docker-build-push * Add job names * Remove build_bench workflow * Update version input description in docker-build-push * Print .env in tests, if version is develop, change to latest (for tag) * Fix env setup * Uncomment tests * Parse and set short tags from git tag in bake file * Move devcontainer settings to devcontainer.json * Add db command notice * Fix CI? * Fix inconsistencies in development readme * Remove pwd for now * Remove custom apps for production instruction * Update todos * Add docs for images and compose files * Add variables docs and allow custom frappe site name header * Add notice about internal environment variables * Update site-operations docs * Update todos * Add Overrides header in images-and-compose-files * Update todos * Remove extra docs * Don't log requests in worker image (nginx already does that) * Remove default value of FRAPPE_SITE_NAME_HEADER in example.env * Use file that consistent in v12, v13 and develop to check /assets * Fix paths in CI * Update todos * Remove TODO.md * Update tests/_check_backup_files.py Co-authored-by: Revant Nandgaonkar <revant.one@gmail.com> * Change variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY to S3_ACCESS_KEY, S3_SECRET_KEY in tests * Fix S3 test * Use `nginxinc/nginx-unprivileged` instead of `nginx` image Also use Ngnix 1.20 instead of unstable 1.21 * Fix https override * Update Dockerfile * Mount assets to backend service in read only mode * Touch .build (#307), use scripts from nginx image to generate config and touch .build * Update example env after building stable images * Touch `.build` on develop image (untill https://github.com/frappe/frappe/issues/15396 is resolved) * Add `make` to worker build deps for linux/arm64 * Fix update example.env job * Fix .build creation on develop branch * Move bench CI to different file This way workflow runs only on PRs that relevant to bench build * Fix app name in custom app example * Update erpnext and frappe versions in example.env * Don't install `svg-sprite` and `sass` node modules in nginx image on linux/arm64 (https://github.com/frappe/frappe/pull/15275) * docs: README and docs * docs: add link to site operations from docker swarm * ci: fix tests as per changes to compose.yaml * docs: move wiki articles to docs * docs: fix add custom domain * docs: fix patch code from images * fix: do not expose port 80 for old images * fix: custom domain labels to frontend container/service * Add missing descriptions to envs in example.env * Fix redis depends_on * Fix docker compose in tests when not running on TTY * Set -T flag in `docker compose exec` only if not tty * Run pre-commit on docs * Remove postgres healthcheck (it gets overriden by mariadb) * Refactor test * Update workflow names * Add pip to dependabot config * docs: backup and push (#19) * Beautify changes by @revant (#20) * feat: add gevent to worker image * feat: real_ip configuration for nginx * Return `healthcheck.sh` just for tests Co-authored-by: Lev Vereshchagin <mail@vrslev.com> * Make pretend bench catch unknown commands (closes #666) * Remove debug print in push-backup * Fix typing issues in push-backup * Update file keys in push-backups: from abs path to <site>/<file> * Refactor push-backup * Move gevent installation in Frappe step * Don't pin boto stubs requirement * Cache pip deps on build * Update example env versions * Refactor check backup files * Fix backup test * Fix backup test * Rename build/ dir to images/ * Rename build/ dir to images/ * Fix /build -> /images in docs * Update example.env * Use reusable workflow in frappe user instead of vrslev * Fix compose`s `project` option in docs (https://github.com/frappe/frappe_docker/pull/617#issuecomment-1065178792) * Add note about project option in site-operations doc * Update example env * Rename build arg `USERNAME` to `REGISTRY_USER` * Allow https proxy to access Docker socket * Revert "Use reusable workflow in frappe user instead of vrslev" This reverts commit6062500d0d. * Revert "Revert "Use reusable workflow in frappe user instead of vrslev"" This reverts commit4680d18ff8. Co-authored-by: Revant Nandgaonkar <revant.one@gmail.com>
This commit is contained in:
Lev
69
tests/_check_backup_files.py
Normal file
69
tests/_check_backup_files.py
Normal file
@@ -0,0 +1,69 @@
|
||||
import os
|
||||
import re
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import boto3
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from mypy_boto3_s3.service_resource import BucketObjectsCollection, _Bucket
|
||||
|
||||
|
||||
def get_bucket() -> "_Bucket":
|
||||
return boto3.resource(
|
||||
service_name="s3",
|
||||
endpoint_url="http://minio:9000",
|
||||
region_name="us-east-1",
|
||||
aws_access_key_id=os.getenv("S3_ACCESS_KEY"),
|
||||
aws_secret_access_key=os.getenv("S3_SECRET_KEY"),
|
||||
).Bucket("frappe")
|
||||
|
||||
|
||||
def get_key_builder():
|
||||
site_name = os.getenv("SITE_NAME")
|
||||
assert site_name
|
||||
|
||||
def builder(key: str, suffix: str) -> bool:
|
||||
return bool(re.match(rf"{site_name}.*{suffix}$", key))
|
||||
|
||||
return builder
|
||||
|
||||
|
||||
def check_keys(objects: "BucketObjectsCollection"):
|
||||
check_key = get_key_builder()
|
||||
|
||||
db = False
|
||||
config = False
|
||||
private_files = False
|
||||
public_files = False
|
||||
|
||||
for obj in objects:
|
||||
if check_key(obj.key, "database.sql.gz"):
|
||||
db = True
|
||||
elif check_key(obj.key, "site_config_backup.json"):
|
||||
config = True
|
||||
elif check_key(obj.key, "private-files.tar"):
|
||||
private_files = True
|
||||
elif check_key(obj.key, "files.tar"):
|
||||
public_files = True
|
||||
|
||||
exc = lambda type_: Exception(f"Didn't push {type_} backup")
|
||||
if not db:
|
||||
raise exc("database")
|
||||
if not config:
|
||||
raise exc("site config")
|
||||
if not private_files:
|
||||
raise exc("private files")
|
||||
if not public_files:
|
||||
raise exc("public files")
|
||||
|
||||
print("All files were pushed to S3!")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
bucket = get_bucket()
|
||||
check_keys(bucket.objects.all())
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
19
tests/_create_bucket.py
Normal file
19
tests/_create_bucket.py
Normal file
@@ -0,0 +1,19 @@
|
||||
import os
|
||||
|
||||
import boto3
|
||||
|
||||
|
||||
def main() -> int:
|
||||
resource = boto3.resource(
|
||||
service_name="s3",
|
||||
endpoint_url="http://minio:9000",
|
||||
region_name="us-east-1",
|
||||
aws_access_key_id=os.getenv("S3_ACCESS_KEY"),
|
||||
aws_secret_access_key=os.getenv("S3_SECRET_KEY"),
|
||||
)
|
||||
resource.create_bucket(Bucket="frappe")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
26
tests/_ping_frappe_connections.py
Normal file
26
tests/_ping_frappe_connections.py
Normal file
@@ -0,0 +1,26 @@
|
||||
import frappe
|
||||
|
||||
|
||||
def check_db():
|
||||
doc = frappe.get_single("System Settings")
|
||||
assert any(v is None for v in doc.as_dict().values()), "Database test didn't pass"
|
||||
print("Database works!")
|
||||
|
||||
|
||||
def check_cache():
|
||||
key_and_name = "mytestkey", "mytestname"
|
||||
frappe.cache().hset(*key_and_name, "mytestvalue")
|
||||
assert frappe.cache().hget(*key_and_name) == "mytestvalue", "Cache test didn't pass"
|
||||
frappe.cache().hdel(*key_and_name)
|
||||
print("Cache works!")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
frappe.connect(site="tests")
|
||||
check_db()
|
||||
check_cache()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
24
tests/compose.ci-erpnext.yaml
Normal file
24
tests/compose.ci-erpnext.yaml
Normal file
@@ -0,0 +1,24 @@
|
||||
services:
|
||||
configurator:
|
||||
image: localhost:5000/frappe/erpnext-worker:${ERPNEXT_VERSION}
|
||||
|
||||
backend:
|
||||
image: localhost:5000/frappe/erpnext-worker:${ERPNEXT_VERSION}
|
||||
|
||||
frontend:
|
||||
image: localhost:5000/frappe/erpnext-nginx:${ERPNEXT_VERSION}
|
||||
|
||||
websocket:
|
||||
image: localhost:5000/frappe/frappe-socketio:${FRAPPE_VERSION}
|
||||
|
||||
queue-short:
|
||||
image: localhost:5000/frappe/erpnext-worker:${ERPNEXT_VERSION}
|
||||
|
||||
queue-default:
|
||||
image: localhost:5000/frappe/erpnext-worker:${ERPNEXT_VERSION}
|
||||
|
||||
queue-long:
|
||||
image: localhost:5000/frappe/erpnext-worker:${ERPNEXT_VERSION}
|
||||
|
||||
scheduler:
|
||||
image: localhost:5000/frappe/erpnext-worker:${ERPNEXT_VERSION}
|
||||
24
tests/compose.ci.yaml
Normal file
24
tests/compose.ci.yaml
Normal file
@@ -0,0 +1,24 @@
|
||||
services:
|
||||
configurator:
|
||||
image: localhost:5000/frappe/frappe-worker:${FRAPPE_VERSION}
|
||||
|
||||
backend:
|
||||
image: localhost:5000/frappe/frappe-worker:${FRAPPE_VERSION}
|
||||
|
||||
frontend:
|
||||
image: localhost:5000/frappe/frappe-nginx:${FRAPPE_VERSION}
|
||||
|
||||
websocket:
|
||||
image: localhost:5000/frappe/frappe-socketio:${FRAPPE_VERSION}
|
||||
|
||||
queue-short:
|
||||
image: localhost:5000/frappe/frappe-worker:${FRAPPE_VERSION}
|
||||
|
||||
queue-default:
|
||||
image: localhost:5000/frappe/frappe-worker:${FRAPPE_VERSION}
|
||||
|
||||
queue-long:
|
||||
image: localhost:5000/frappe/frappe-worker:${FRAPPE_VERSION}
|
||||
|
||||
scheduler:
|
||||
image: localhost:5000/frappe/frappe-worker:${FRAPPE_VERSION}
|
||||
@@ -1,26 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
print_group() {
|
||||
echo ::endgroup::
|
||||
echo "::group::$*"
|
||||
}
|
||||
|
||||
ping_site() {
|
||||
print_group "Ping site $SITE_NAME"
|
||||
|
||||
echo Ping version
|
||||
ping_res=$(curl -sS "http://$SITE_NAME/api/method/version")
|
||||
echo "$ping_res"
|
||||
if [[ -z $(echo "$ping_res" | grep "message" || echo "") ]]; then
|
||||
echo "Ping failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo Check index
|
||||
index_res=$(curl -sS "http://$SITE_NAME")
|
||||
if [[ -n $(echo "$index_res" | grep "Internal Server Error" || echo "") ]]; then
|
||||
echo "Index check failed"
|
||||
echo "$index_res"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
28
tests/healthcheck.sh
Executable file
28
tests/healthcheck.sh
Executable file
@@ -0,0 +1,28 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
get_key() {
|
||||
jq -r ".$1" /home/frappe/frappe-bench/sites/common_site_config.json
|
||||
}
|
||||
|
||||
get_redis_url() {
|
||||
URL=$(get_key "$1" | sed 's|redis://||g')
|
||||
if [[ ${URL} == *"/"* ]]; then
|
||||
URL=$(echo "${URL}" | cut -f1 -d"/")
|
||||
fi
|
||||
echo "$URL"
|
||||
}
|
||||
|
||||
check_connection() {
|
||||
echo "Check $1"
|
||||
wait-for-it "$1" -t 1
|
||||
}
|
||||
|
||||
check_connection "$(get_key db_host):$(get_key db_port)"
|
||||
check_connection "$(get_redis_url redis_cache)"
|
||||
check_connection "$(get_redis_url redis_queue)"
|
||||
check_connection "$(get_redis_url redis_socketio)"
|
||||
|
||||
if [[ "$1" = -p || "$1" = --ping-service ]]; then
|
||||
check_connection "$2"
|
||||
fi
|
||||
@@ -1,283 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
source tests/functions.sh
|
||||
|
||||
project_name=frappe_bench_00
|
||||
|
||||
docker_compose_with_args() {
|
||||
# shellcheck disable=SC2068
|
||||
docker-compose \
|
||||
-p $project_name \
|
||||
-f installation/docker-compose-common.yml \
|
||||
-f installation/docker-compose-frappe.yml \
|
||||
-f installation/frappe-publish.yml \
|
||||
$@
|
||||
}
|
||||
|
||||
check_migration_complete() {
|
||||
print_group Check migration
|
||||
|
||||
container_id=$(docker_compose_with_args ps -q frappe-python)
|
||||
cmd="docker logs ${container_id} 2>&1 | grep 'Starting gunicorn' || echo ''"
|
||||
worker_log=$(eval "$cmd")
|
||||
INCREMENT=0
|
||||
|
||||
while [[ ${worker_log} != *"Starting gunicorn"* && ${INCREMENT} -lt 120 ]]; do
|
||||
sleep 3
|
||||
((INCREMENT = INCREMENT + 1))
|
||||
echo "Wait for migration to complete..."
|
||||
worker_log=$(eval "$cmd")
|
||||
if [[ ${worker_log} != *"Starting gunicorn"* && ${INCREMENT} -eq 120 ]]; then
|
||||
echo Migration timeout
|
||||
docker logs "${container_id}"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo Migration Log
|
||||
docker logs "${container_id}"
|
||||
}
|
||||
|
||||
check_health() {
|
||||
print_group Loop health check
|
||||
|
||||
docker run --name frappe_doctor \
|
||||
-v "${project_name}_sites-vol:/home/frappe/frappe-bench/sites" \
|
||||
--network "${project_name}_default" \
|
||||
frappe/frappe-worker:edge doctor || true
|
||||
|
||||
cmd='docker logs frappe_doctor | grep "Health check successful" || echo ""'
|
||||
doctor_log=$(eval "$cmd")
|
||||
INCREMENT=0
|
||||
|
||||
while [[ -z "${doctor_log}" && ${INCREMENT} -lt 60 ]]; do
|
||||
sleep 1
|
||||
((INCREMENT = INCREMENT + 1))
|
||||
container=$(docker start frappe_doctor)
|
||||
echo "Restarting ${container}..."
|
||||
doctor_log=$(eval "$cmd")
|
||||
|
||||
if [[ ${INCREMENT} -eq 60 ]]; then
|
||||
docker logs "${container}"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Initial group
|
||||
echo ::group::Setup .env
|
||||
cp env-example .env
|
||||
sed -i -e "s/edge/v13/g" .env
|
||||
cat .env
|
||||
# shellcheck disable=SC2046
|
||||
export $(cat .env)
|
||||
|
||||
print_group Start services
|
||||
echo Start main services
|
||||
docker_compose_with_args up -d --quiet-pull
|
||||
|
||||
echo Start postgres
|
||||
docker pull postgres:11.8 -q
|
||||
docker run \
|
||||
--name postgresql \
|
||||
-d \
|
||||
-e POSTGRES_PASSWORD=admin \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
postgres:11.8
|
||||
|
||||
check_health
|
||||
|
||||
print_group "Create new site "
|
||||
SITE_NAME=test.localhost
|
||||
docker run \
|
||||
--rm \
|
||||
-e SITE_NAME=$SITE_NAME \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:v13 new
|
||||
|
||||
ping_site
|
||||
|
||||
print_group "Update .env (v13 -> edge)"
|
||||
sed -i -e "s/v13/edge/g" .env
|
||||
cat .env
|
||||
# shellcheck disable=SC2046
|
||||
export $(cat .env)
|
||||
|
||||
print_group Restart containers
|
||||
docker_compose_with_args stop
|
||||
docker_compose_with_args up -d
|
||||
|
||||
check_migration_complete
|
||||
sleep 5
|
||||
ping_site
|
||||
|
||||
PG_SITE_NAME=pgsql.localhost
|
||||
print_group "Create new site (Postgres)"
|
||||
docker run \
|
||||
--rm \
|
||||
-e SITE_NAME=$PG_SITE_NAME \
|
||||
-e POSTGRES_HOST=postgresql \
|
||||
-e DB_ROOT_USER=postgres \
|
||||
-e POSTGRES_PASSWORD=admin \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge new
|
||||
|
||||
check_migration_complete
|
||||
SITE_NAME=$PG_SITE_NAME ping_site
|
||||
|
||||
print_group Backup site
|
||||
docker run \
|
||||
--rm \
|
||||
-e WITH_FILES=1 \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge backup
|
||||
|
||||
MINIO_ACCESS_KEY="AKIAIOSFODNN7EXAMPLE"
|
||||
MINIO_SECRET_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
|
||||
|
||||
print_group Prepare S3 server
|
||||
echo Start S3 server
|
||||
docker run \
|
||||
--name minio \
|
||||
-d \
|
||||
-e "MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY" \
|
||||
-e "MINIO_SECRET_KEY=$MINIO_SECRET_KEY" \
|
||||
--network ${project_name}_default \
|
||||
minio/minio server /data
|
||||
|
||||
echo Create bucket
|
||||
docker run \
|
||||
--rm \
|
||||
--network ${project_name}_default \
|
||||
vltgroup/s3cmd:latest \
|
||||
s3cmd \
|
||||
--access_key=$MINIO_ACCESS_KEY \
|
||||
--secret_key=$MINIO_SECRET_KEY \
|
||||
--region=us-east-1 \
|
||||
--no-ssl \
|
||||
--host=minio:9000 \
|
||||
--host-bucket=minio:9000 \
|
||||
mb s3://frappe
|
||||
|
||||
print_group Push backup
|
||||
docker run \
|
||||
--rm \
|
||||
-e BUCKET_NAME=frappe \
|
||||
-e REGION=us-east-1 \
|
||||
-e BUCKET_DIR=local \
|
||||
-e ACCESS_KEY_ID=$MINIO_ACCESS_KEY \
|
||||
-e SECRET_ACCESS_KEY=$MINIO_SECRET_KEY \
|
||||
-e ENDPOINT_URL=http://minio:9000 \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge push-backup
|
||||
|
||||
print_group Prune and restart services
|
||||
docker_compose_with_args stop
|
||||
docker container prune -f && docker volume prune -f
|
||||
docker_compose_with_args up -d
|
||||
|
||||
check_health
|
||||
|
||||
print_group Restore backup from S3
|
||||
docker run \
|
||||
--rm \
|
||||
-e MYSQL_ROOT_PASSWORD=admin \
|
||||
-e BUCKET_NAME=frappe \
|
||||
-e BUCKET_DIR=local \
|
||||
-e ACCESS_KEY_ID=$MINIO_ACCESS_KEY \
|
||||
-e SECRET_ACCESS_KEY=$MINIO_SECRET_KEY \
|
||||
-e ENDPOINT_URL=http://minio:9000 \
|
||||
-e REGION=us-east-1 \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge restore-backup
|
||||
|
||||
check_health
|
||||
ping_site
|
||||
SITE_NAME=$PG_SITE_NAME ping_site
|
||||
|
||||
EDGE_SITE_NAME=edge.localhost
|
||||
print_group "Create new site (edge)"
|
||||
docker run \
|
||||
--rm \
|
||||
-e SITE_NAME=$EDGE_SITE_NAME \
|
||||
-e INSTALL_APPS=frappe \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge new
|
||||
|
||||
check_health
|
||||
SITE_NAME=$EDGE_SITE_NAME ping_site
|
||||
|
||||
print_group Migrate edge site
|
||||
docker run \
|
||||
--rm \
|
||||
-e MAINTENANCE_MODE=1 \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
-v ${project_name}_assets-vol:/home/frappe/frappe-bench/sites/assets \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge migrate
|
||||
|
||||
check_migration_complete
|
||||
|
||||
print_group "Restore backup S3 (overwrite)"
|
||||
docker run \
|
||||
--rm \
|
||||
-e MYSQL_ROOT_PASSWORD=admin \
|
||||
-e BUCKET_NAME=frappe \
|
||||
-e BUCKET_DIR=local \
|
||||
-e ACCESS_KEY_ID=$MINIO_ACCESS_KEY \
|
||||
-e SECRET_ACCESS_KEY=$MINIO_SECRET_KEY \
|
||||
-e ENDPOINT_URL=http://minio:9000 \
|
||||
-e REGION=us-east-1 \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge restore-backup
|
||||
|
||||
check_migration_complete
|
||||
ping_site
|
||||
|
||||
print_group "Check console for $SITE_NAME"
|
||||
docker run \
|
||||
--rm \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge console $SITE_NAME
|
||||
|
||||
print_group "Check console for $PG_SITE_NAME"
|
||||
docker run \
|
||||
--rm \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge console $PG_SITE_NAME
|
||||
|
||||
print_group "Check drop site for $SITE_NAME (MariaDB)"
|
||||
docker run \
|
||||
--rm \
|
||||
-e SITE_NAME=$SITE_NAME \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge drop
|
||||
|
||||
print_group "Check drop site for $PG_SITE_NAME (Postgres)"
|
||||
docker run \
|
||||
--rm \
|
||||
-e SITE_NAME=$PG_SITE_NAME \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:edge drop
|
||||
|
||||
print_group Check bench --help
|
||||
docker run \
|
||||
--rm \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
--user frappe \
|
||||
frappe/frappe-worker:edge bench --help
|
||||
502
tests/main.py
Normal file
502
tests/main.py
Normal file
@@ -0,0 +1,502 @@
|
||||
import os
|
||||
import shlex
|
||||
import shutil
|
||||
import ssl
|
||||
import subprocess
|
||||
import sys
|
||||
from enum import Enum
|
||||
from functools import wraps
|
||||
from textwrap import dedent
|
||||
from time import sleep
|
||||
from typing import Any, Callable, Optional
|
||||
from urllib.error import HTTPError, URLError
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
CI = os.getenv("CI")
|
||||
SITE_NAME = "tests"
|
||||
BACKEND_SERVICES = (
|
||||
"backend",
|
||||
"queue-short",
|
||||
"queue-default",
|
||||
"queue-long",
|
||||
"scheduler",
|
||||
)
|
||||
S3_ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE"
|
||||
S3_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
|
||||
TTY = sys.stdout.isatty()
|
||||
|
||||
|
||||
def patch_print():
|
||||
# Patch `print()` builtin to have nice logs when running GitHub Actions
|
||||
if not CI:
|
||||
return
|
||||
global print
|
||||
_old_print = print
|
||||
|
||||
def print(
|
||||
*values: Any,
|
||||
sep: Optional[str] = None,
|
||||
end: Optional[str] = None,
|
||||
file: Any = None,
|
||||
flush: bool = False,
|
||||
):
|
||||
return _old_print(*values, sep=sep, end=end, file=file, flush=True)
|
||||
|
||||
|
||||
class Color(Enum):
|
||||
GREY = 30
|
||||
RED = 31
|
||||
GREEN = 32
|
||||
YELLOW = 33
|
||||
BLUE = 34
|
||||
MAGENTA = 35
|
||||
CYAN = 36
|
||||
WHITE = 37
|
||||
|
||||
|
||||
def colored(text: str, color: Color):
|
||||
return f"\033[{color.value}m{text}\033[0m"
|
||||
|
||||
|
||||
def log(text: str):
|
||||
def decorator(f: Callable[..., Any]):
|
||||
@wraps(f)
|
||||
def wrapper(*args: Any, **kwargs: Any):
|
||||
if CI:
|
||||
print(f"::group::{text}")
|
||||
else:
|
||||
output = (
|
||||
f"\n{f' {text} '.center(os.get_terminal_size().columns, '=')}\n"
|
||||
)
|
||||
print(colored(output, Color.YELLOW))
|
||||
|
||||
ret = f(*args, **kwargs)
|
||||
|
||||
if CI:
|
||||
print("::endgroup::")
|
||||
return ret
|
||||
|
||||
return wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
def run(*cmd: str):
|
||||
print(colored(f"> {shlex.join(cmd)}", Color.GREEN))
|
||||
return subprocess.check_call(cmd)
|
||||
|
||||
|
||||
def docker_compose(*cmd: str):
|
||||
args = [
|
||||
"docker",
|
||||
"compose",
|
||||
"-p",
|
||||
"test",
|
||||
"--env-file",
|
||||
"tests/.env",
|
||||
"-f",
|
||||
"compose.yaml",
|
||||
"-f",
|
||||
"overrides/compose.proxy.yaml",
|
||||
"-f",
|
||||
"overrides/compose.mariadb.yaml",
|
||||
"-f",
|
||||
"overrides/compose.redis.yaml",
|
||||
]
|
||||
if CI:
|
||||
args.extend(("-f", "tests/compose.ci.yaml"))
|
||||
return run(*args, *cmd)
|
||||
|
||||
|
||||
def docker_compose_exec(*cmd: str):
|
||||
if TTY:
|
||||
return docker_compose("exec", *cmd)
|
||||
else:
|
||||
return docker_compose("exec", "-T", *cmd)
|
||||
|
||||
|
||||
@log("Setup .env")
|
||||
def setup_env():
|
||||
shutil.copy("example.env", "tests/.env")
|
||||
|
||||
if not CI:
|
||||
return
|
||||
|
||||
for var in ("FRAPPE_VERSION", "ERPNEXT_VERSION"):
|
||||
if os.environ[var] == "develop":
|
||||
with open(os.environ["GITHUB_ENV"], "a") as f:
|
||||
f.write(f"\n{var}=latest")
|
||||
os.environ[var] = "latest"
|
||||
|
||||
with open("tests/.env", "a") as f:
|
||||
f.write(
|
||||
dedent(
|
||||
f"""
|
||||
FRAPPE_VERSION={os.environ['FRAPPE_VERSION']}
|
||||
ERPNEXT_VERSION={os.environ['ERPNEXT_VERSION']}
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
with open("tests/.env") as f:
|
||||
print(f.read())
|
||||
|
||||
|
||||
@log("Print configuration")
|
||||
def print_compose_configuration():
|
||||
docker_compose("config")
|
||||
|
||||
|
||||
@log("Create containers")
|
||||
def create_containers():
|
||||
docker_compose("up", "-d", "--quiet-pull")
|
||||
|
||||
|
||||
@log("Check if Python services have connections")
|
||||
def ping_links_in_backends():
|
||||
for service in BACKEND_SERVICES:
|
||||
docker_compose("cp", "tests/healthcheck.sh", f"{service}:/tmp/")
|
||||
for _ in range(10):
|
||||
try:
|
||||
docker_compose_exec(service, "bash", "/tmp/healthcheck.sh")
|
||||
break
|
||||
except subprocess.CalledProcessError:
|
||||
sleep(1)
|
||||
else:
|
||||
raise RuntimeError(f"Connections healthcheck failed for service {service}")
|
||||
|
||||
|
||||
@log("Create site")
|
||||
def create_site():
|
||||
docker_compose_exec(
|
||||
"backend",
|
||||
"bench",
|
||||
"new-site",
|
||||
SITE_NAME,
|
||||
"--mariadb-root-password",
|
||||
"123",
|
||||
"--admin-password",
|
||||
"admin",
|
||||
)
|
||||
docker_compose("restart", "backend")
|
||||
|
||||
|
||||
# This is needed to check https override
|
||||
_ssl_ctx = ssl.create_default_context()
|
||||
_ssl_ctx.check_hostname = False
|
||||
_ssl_ctx.verify_mode = ssl.CERT_NONE
|
||||
|
||||
|
||||
def ping_and_check_content(url: str, callback: Callable[[str], Optional[str]]):
|
||||
request = Request(url, headers={"Host": SITE_NAME})
|
||||
print(f"Checking {url}")
|
||||
|
||||
for _ in range(100):
|
||||
try:
|
||||
response = urlopen(request, context=_ssl_ctx)
|
||||
|
||||
except HTTPError as exc:
|
||||
if exc.code not in (404, 502):
|
||||
raise
|
||||
|
||||
except URLError:
|
||||
pass
|
||||
|
||||
else:
|
||||
text: str = response.read().decode()
|
||||
ret = callback(text)
|
||||
if ret:
|
||||
print(ret)
|
||||
return
|
||||
|
||||
sleep(0.1)
|
||||
|
||||
raise RuntimeError(f"Couldn't ping {url}")
|
||||
|
||||
|
||||
def check_index_callback(text: str):
|
||||
if "404 page not found" not in text:
|
||||
return text[:200]
|
||||
|
||||
|
||||
@log("Check /")
|
||||
def check_index():
|
||||
ping_and_check_content("http://127.0.0.1", check_index_callback)
|
||||
|
||||
|
||||
@log("Check /api/method/version")
|
||||
def check_api():
|
||||
ping_and_check_content(
|
||||
"http://127.0.0.1/api/method/version",
|
||||
lambda text: text if '"message"' in text else None,
|
||||
)
|
||||
|
||||
|
||||
@log("Check if Frappe can connect to services in Python services")
|
||||
def ping_frappe_connections_in_backends():
|
||||
for service in BACKEND_SERVICES:
|
||||
docker_compose("cp", "tests/_ping_frappe_connections.py", f"{service}:/tmp/")
|
||||
docker_compose_exec(
|
||||
service,
|
||||
"/home/frappe/frappe-bench/env/bin/python",
|
||||
f"/tmp/_ping_frappe_connections.py",
|
||||
)
|
||||
|
||||
|
||||
@log("Check /assets")
|
||||
def check_assets():
|
||||
ping_and_check_content(
|
||||
"http://127.0.0.1/assets/frappe/images/frappe-framework-logo.svg",
|
||||
lambda text: text[:200] if text else None,
|
||||
)
|
||||
|
||||
|
||||
@log("Check /files")
|
||||
def check_files():
|
||||
file_name = "testfile.txt"
|
||||
docker_compose(
|
||||
"cp",
|
||||
f"tests/{file_name}",
|
||||
f"backend:/home/frappe/frappe-bench/sites/{SITE_NAME}/public/files/",
|
||||
)
|
||||
ping_and_check_content(
|
||||
f"http://127.0.0.1/files/{file_name}",
|
||||
lambda text: text if text == "lalala\n" else None,
|
||||
)
|
||||
|
||||
|
||||
@log("Prepare S3 server")
|
||||
def prepare_s3_server():
|
||||
run(
|
||||
"docker",
|
||||
"run",
|
||||
"--name",
|
||||
"minio",
|
||||
"-d",
|
||||
"-e",
|
||||
f"MINIO_ACCESS_KEY={S3_ACCESS_KEY}",
|
||||
"-e",
|
||||
f"MINIO_SECRET_KEY={S3_SECRET_KEY}",
|
||||
"--network",
|
||||
"test_default",
|
||||
"minio/minio",
|
||||
"server",
|
||||
"/data",
|
||||
)
|
||||
docker_compose("cp", "tests/_create_bucket.py", "backend:/tmp")
|
||||
docker_compose_exec(
|
||||
"-e",
|
||||
f"S3_ACCESS_KEY={S3_ACCESS_KEY}",
|
||||
"-e",
|
||||
f"S3_SECRET_KEY={S3_SECRET_KEY}",
|
||||
"backend",
|
||||
"/home/frappe/frappe-bench/env/bin/python",
|
||||
"/tmp/_create_bucket.py",
|
||||
)
|
||||
|
||||
|
||||
@log("Push backup to S3")
|
||||
def push_backup_to_s3():
|
||||
docker_compose_exec(
|
||||
"backend", "bench", "--site", SITE_NAME, "backup", "--with-files"
|
||||
)
|
||||
docker_compose_exec(
|
||||
"backend",
|
||||
"push-backup",
|
||||
"--site",
|
||||
SITE_NAME,
|
||||
"--bucket",
|
||||
"frappe",
|
||||
"--region-name",
|
||||
"us-east-1",
|
||||
"--endpoint-url",
|
||||
"http://minio:9000",
|
||||
"--aws-access-key-id",
|
||||
S3_ACCESS_KEY,
|
||||
"--aws-secret-access-key",
|
||||
S3_SECRET_KEY,
|
||||
)
|
||||
|
||||
|
||||
@log("Check backup files in S3")
|
||||
def check_backup_in_s3():
|
||||
docker_compose("cp", "tests/_check_backup_files.py", "backend:/tmp")
|
||||
docker_compose_exec(
|
||||
"-e",
|
||||
f"S3_ACCESS_KEY={S3_ACCESS_KEY}",
|
||||
"-e",
|
||||
f"S3_SECRET_KEY={S3_SECRET_KEY}",
|
||||
"-e",
|
||||
f"SITE_NAME={SITE_NAME}",
|
||||
"backend",
|
||||
"/home/frappe/frappe-bench/env/bin/python",
|
||||
"/tmp/_check_backup_files.py",
|
||||
)
|
||||
|
||||
|
||||
@log("Stop S3 container")
|
||||
def stop_s3_container():
|
||||
run("docker", "rm", "minio", "-f")
|
||||
|
||||
|
||||
@log("Recreate with HTTPS override")
|
||||
def recreate_with_https_override():
|
||||
docker_compose("-f", "overrides/compose.https.yaml", "up", "-d")
|
||||
|
||||
|
||||
@log("Check / (HTTPS)")
|
||||
def check_index_https():
|
||||
ping_and_check_content("https://127.0.0.1", check_index_callback)
|
||||
|
||||
|
||||
@log("Stop containers")
|
||||
def stop_containers():
|
||||
docker_compose("down", "-v", "--remove-orphans")
|
||||
|
||||
|
||||
@log("Recreate with ERPNext override")
|
||||
def create_containers_with_erpnext_override():
|
||||
args = ["-f", "overrides/compose.erpnext.yaml"]
|
||||
if CI:
|
||||
args += ("-f", "tests/compose.ci-erpnext.yaml")
|
||||
|
||||
docker_compose(*args, "up", "-d", "--quiet-pull")
|
||||
|
||||
|
||||
@log("Create ERPNext site")
|
||||
def create_erpnext_site():
|
||||
docker_compose_exec(
|
||||
"backend",
|
||||
"bench",
|
||||
"new-site",
|
||||
SITE_NAME,
|
||||
"--mariadb-root-password",
|
||||
"123",
|
||||
"--admin-password",
|
||||
"admin",
|
||||
"--install-app",
|
||||
"erpnext",
|
||||
)
|
||||
docker_compose("restart", "backend")
|
||||
|
||||
|
||||
@log("Check /api/method")
|
||||
def check_erpnext_api():
|
||||
ping_and_check_content(
|
||||
"http://127.0.0.1/api/method/erpnext.templates.pages.product_search.get_product_list",
|
||||
lambda text: text if '"message"' in text else None,
|
||||
)
|
||||
|
||||
|
||||
@log("Check /assets")
|
||||
def check_erpnext_assets():
|
||||
ping_and_check_content(
|
||||
"http://127.0.0.1/assets/erpnext/js/setup_wizard.js",
|
||||
lambda text: text[:200] if text else None,
|
||||
)
|
||||
|
||||
|
||||
@log("Create containers with Postgres override")
|
||||
def create_containers_with_postgres_override():
|
||||
docker_compose("-f", "overrides/compose.postgres.yaml", "up", "-d", "--quiet-pull")
|
||||
|
||||
|
||||
@log("Create Postgres site")
|
||||
def create_postgres_site():
|
||||
docker_compose_exec(
|
||||
"backend", "bench", "set-config", "-g", "root_login", "postgres"
|
||||
)
|
||||
docker_compose_exec("backend", "bench", "set-config", "-g", "root_password", "123")
|
||||
docker_compose_exec(
|
||||
"backend",
|
||||
"bench",
|
||||
"new-site",
|
||||
SITE_NAME,
|
||||
"--db-type",
|
||||
"postgres",
|
||||
"--admin-password",
|
||||
"admin",
|
||||
)
|
||||
docker_compose("restart", "backend")
|
||||
|
||||
|
||||
@log("Delete .env")
|
||||
def delete_env():
|
||||
os.remove("tests/.env")
|
||||
|
||||
|
||||
@log("Show logs")
|
||||
def show_docker_compose_logs():
|
||||
docker_compose("logs")
|
||||
|
||||
|
||||
def start():
|
||||
setup_env()
|
||||
print_compose_configuration()
|
||||
create_containers()
|
||||
ping_links_in_backends()
|
||||
|
||||
|
||||
def create_frappe_site_and_check_availability():
|
||||
create_site()
|
||||
check_index()
|
||||
check_api()
|
||||
ping_frappe_connections_in_backends()
|
||||
check_assets()
|
||||
check_files()
|
||||
|
||||
|
||||
def check_s3():
|
||||
prepare_s3_server()
|
||||
|
||||
try:
|
||||
push_backup_to_s3()
|
||||
check_backup_in_s3()
|
||||
finally:
|
||||
stop_s3_container()
|
||||
|
||||
|
||||
def check_https():
|
||||
print_compose_configuration()
|
||||
recreate_with_https_override()
|
||||
check_index_https()
|
||||
stop_containers()
|
||||
|
||||
|
||||
def check_erpnext():
|
||||
print_compose_configuration()
|
||||
create_containers_with_erpnext_override()
|
||||
create_erpnext_site()
|
||||
check_erpnext_api()
|
||||
check_erpnext_assets()
|
||||
stop_containers()
|
||||
|
||||
|
||||
def check_postgres():
|
||||
print_compose_configuration()
|
||||
create_containers_with_postgres_override()
|
||||
create_postgres_site()
|
||||
ping_links_in_backends()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
patch_print()
|
||||
start()
|
||||
create_frappe_site_and_check_availability()
|
||||
check_s3()
|
||||
check_https()
|
||||
check_erpnext()
|
||||
check_postgres()
|
||||
|
||||
finally:
|
||||
delete_env()
|
||||
show_docker_compose_logs()
|
||||
stop_containers()
|
||||
|
||||
print(colored("\nTests successfully passed!", Color.YELLOW))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,44 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
source tests/functions.sh
|
||||
|
||||
project_name="test_erpnext"
|
||||
SITE_NAME="test_erpnext.localhost"
|
||||
|
||||
echo ::group::Setup env
|
||||
cp env-example .env
|
||||
sed -i -e "s/edge/test/g" .env
|
||||
# shellcheck disable=SC2046
|
||||
export $(cat .env)
|
||||
cat .env
|
||||
|
||||
print_group Start services
|
||||
docker-compose \
|
||||
-p $project_name \
|
||||
-f installation/docker-compose-common.yml \
|
||||
-f installation/docker-compose-erpnext.yml \
|
||||
-f installation/erpnext-publish.yml \
|
||||
up -d
|
||||
|
||||
print_group Fix permissions
|
||||
docker run \
|
||||
--rm \
|
||||
--user root \
|
||||
-v ${project_name}_sites-vol:/sites \
|
||||
-v ${project_name}_assets-vol:/assets \
|
||||
-v ${project_name}_logs-vol:/logs \
|
||||
frappe/erpnext-worker:test chown -R 1000:1000 /logs /sites /assets
|
||||
|
||||
print_group Create site
|
||||
docker run \
|
||||
--rm \
|
||||
-e "SITE_NAME=$SITE_NAME" \
|
||||
-e "INSTALL_APPS=erpnext" \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/erpnext-worker:test new
|
||||
|
||||
ping_site
|
||||
rm .env
|
||||
@@ -1,43 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
source tests/functions.sh
|
||||
|
||||
project_name="test_frappe"
|
||||
SITE_NAME="test_frappe.localhost"
|
||||
|
||||
docker_compose_with_args() {
|
||||
# shellcheck disable=SC2068
|
||||
docker-compose \
|
||||
-p $project_name \
|
||||
-f installation/docker-compose-common.yml \
|
||||
-f installation/docker-compose-frappe.yml \
|
||||
-f installation/frappe-publish.yml \
|
||||
$@
|
||||
}
|
||||
|
||||
echo ::group::Setup env
|
||||
cp env-example .env
|
||||
sed -i -e "s/edge/test/g" .env
|
||||
# shellcheck disable=SC2046
|
||||
export $(cat .env)
|
||||
cat .env
|
||||
|
||||
print_group Start services
|
||||
docker_compose_with_args up -d
|
||||
|
||||
print_group Create site
|
||||
docker run \
|
||||
--rm \
|
||||
-e "SITE_NAME=$SITE_NAME" \
|
||||
-v ${project_name}_sites-vol:/home/frappe/frappe-bench/sites \
|
||||
--network ${project_name}_default \
|
||||
frappe/frappe-worker:test new
|
||||
|
||||
ping_site
|
||||
|
||||
print_group Stop and remove containers
|
||||
docker_compose_with_args down
|
||||
|
||||
rm .env
|
||||
1
tests/testfile.txt
Normal file
1
tests/testfile.txt
Normal file
@@ -0,0 +1 @@
|
||||
lalala
|
||||
Reference in New Issue
Block a user