admin/frappe_docker
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

Global refactoring (#617)

Browse Source 
* Rename `bench-build` target to `bench` in bake file

* Update bake file and break everything

* Rename docker-compose.yml to compose.yml to avoid conflicting on `docker buildx bake`

* Fix groups in bake file

* Update frappe-worker

* Update frappe-nginx, erpnext-nginx

* Remove old erpnext images

* Update frappe-socketio

* Fix develop frappe-nginx build on linux/arm64

* Update dockerignore

* Update gitignore

* Update gitignore

* Update .env files

* Update installation (overrides)

* Update tests

* Fix image names

* Update compose

* Update get-latest-tags

* Update CI

* Setup and remove .env on tests

* Add build bench workflow

* Add triggers to main workflow

* Add release helm job

* Use reusable workflows

* Rollback

* Print configuration before running tests

* Show tests/.env

* Revert "Show tests/.env"

This reverts commit 4bc3bdebaf.

* Fix ci image versions

* Remove `frappe-` prefix in build directories

* Move requirements-dev.txt

* Fix image name in CI

* Update gitignore

* Update pre-commit config

* Drop `version:` in compose files

* Add push-backup

* Fix postgres CI test

* Change .yml to .yaml in compose file to follow compose-spec

* Remove prettierignore

* Fix dockerignore

* Change .yml to .yaml in compose file to follow compose-spec

* Don't depend on boto3 while testing (do it in backend)

* Update erpnext example version

* Don't fail ping on URLError

* Move assets volume to main compose file

* Fix type annotations for v12

* Fix postgres ci override in tests

* Fix spaces in socketio

* Reorder stages in nginx image, improve perfomance

* Remove unused todo

* Optimize worker build

* Install Node in worker image

* Add 502 error page

* Remove unused quiet-pull in tests

* Add configurator service to dynamically set common config

* Remove unused compose.ci-postgres.yml

* Use Python for configurator service: faster and more robust

* Add TODO.md

* Use python script to get latest tags in CI

* Clean up nginx dockerfile

* Remove VOLUME declaration

https://stackoverflow.com/a/55052682

* Add custom app example

* Remove pwd for now

* Remove pwd for now

* Use jq for parsing config in healthcheck

* Take advantage of yaml lang: add defaults in compose file. Also require env vars

* Fix CI

* Use resusable workflow

* Update

* Move release_helm job to main.yml

* Rename docker-build to docker-build-push

* Rename main to build_stable

* Rename bench targets

* Remove quotes from docker-build-push inputs

* Update build develop

* Remove HELM_DEPLOY_KEY secret from docker-build-push

* Add job names

* Remove build_bench workflow

* Update version input description in docker-build-push

* Print .env in tests, if version is develop, change to latest (for tag)

* Fix env setup

* Uncomment tests

* Parse and set short tags from git tag in bake file

* Move devcontainer settings to devcontainer.json

* Add db command notice

* Fix CI?

* Fix inconsistencies in development readme

* Remove pwd for now

* Remove custom apps for production instruction

* Update todos

* Add docs for images and compose files

* Add variables docs and allow custom frappe site name header

* Add notice about internal environment variables

* Update site-operations docs

* Update todos

* Add Overrides header in images-and-compose-files

* Update todos

* Remove extra docs

* Don't log requests in worker image (nginx already does that)

* Remove default value of FRAPPE_SITE_NAME_HEADER in example.env

* Use file that consistent in v12, v13 and develop to check /assets

* Fix paths in CI

* Update todos

* Remove TODO.md

* Update tests/_check_backup_files.py

Co-authored-by: Revant Nandgaonkar <revant.one@gmail.com>

* Change variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY to S3_ACCESS_KEY, S3_SECRET_KEY in tests

* Fix S3 test

* Use `nginxinc/nginx-unprivileged` instead of `nginx` image

Also use Ngnix 1.20 instead of unstable 1.21

* Fix https override

* Update Dockerfile

* Mount assets to backend service in read only mode

* Touch .build (#307), use scripts from nginx image to generate config and touch .build

* Update example env after building stable images

* Touch `.build` on develop image (untill https://github.com/frappe/frappe/issues/15396 is resolved)

* Add `make` to worker build deps for linux/arm64

* Fix update example.env job

* Fix .build creation on develop branch

* Move bench CI to different file

This way workflow runs only on PRs that relevant to bench build

* Fix app name in custom app example

* Update erpnext and frappe versions in example.env

* Don't install `svg-sprite` and `sass` node modules in nginx image on linux/arm64 (https://github.com/frappe/frappe/pull/15275)

* docs: README and docs

* docs: add link to site operations from docker swarm

* ci: fix tests as per changes to compose.yaml

* docs: move wiki articles to docs

* docs: fix add custom domain

* docs: fix patch code from images

* fix: do not expose port 80 for old images

* fix: custom domain labels to frontend container/service

* Add missing descriptions to envs in example.env

* Fix redis depends_on

* Fix docker compose in tests when not running on TTY

* Set -T flag in `docker compose exec` only if not tty

* Run pre-commit on docs

* Remove postgres healthcheck (it gets overriden by mariadb)

* Refactor test

* Update workflow names

* Add pip to dependabot config

* docs: backup and push (#19)

* Beautify changes by @revant (#20)

* feat: add gevent to worker image

* feat: real_ip configuration for nginx

* Return `healthcheck.sh` just for tests

Co-authored-by: Lev Vereshchagin <mail@vrslev.com>

* Make pretend bench catch unknown commands (closes #666)

* Remove debug print in push-backup

* Fix typing issues in push-backup

* Update file keys in push-backups: from abs path to <site>/<file>

* Refactor push-backup

* Move gevent installation in Frappe step

* Don't pin boto stubs requirement

* Cache pip deps on build

* Update example env versions

* Refactor check backup files

* Fix backup test

* Fix backup test

* Rename build/ dir to images/

* Rename build/ dir to images/

* Fix /build -> /images in docs

* Update example.env

* Use reusable workflow in frappe user instead of vrslev

* Fix compose`s `project` option in docs (https://github.com/frappe/frappe_docker/pull/617#issuecomment-1065178792)

* Add note about project option in site-operations doc

* Update example env

* Rename build arg `USERNAME` to `REGISTRY_USER`

* Allow https proxy to access Docker socket

* Revert "Use reusable workflow in frappe user instead of vrslev"

This reverts commit 6062500d0d.

* Revert "Revert "Use reusable workflow in frappe user instead of vrslev""

This reverts commit 4680d18ff8.

Co-authored-by: Revant Nandgaonkar <revant.one@gmail.com>
This commit is contained in:
Lev
2022-03-14 08:53:03 +03:00
committed by GitHub
parent 95aeb32e2d
commit a9b6b755ce
116 changed files with 2574 additions and 4975 deletions
Download Patch File Download Diff File Expand all files Collapse all files

258
docs/site-operations.md
View File

@@ -1,177 +1,41 @@
# Site operations
Create and use env file to pass environment variables to containers,
> 💡 You should setup `--project-name` option in `docker-compose` commands if you have non-standard project name.
```sh
source .env
```
Or specify environment variables instead of passing secrets as command arguments. Refer notes section for environment variables required
## Setup New Site
## Setup new site
Note:
- Wait for the database service to start before trying to create a new site.
- If new site creation fails, retry after the MariaDB container is up and running.
- If you're using a managed database instance, make sure that the database is running before setting up a new site.
#### MariaDB Site
- Wait for the `db` service to start and `configurator` to exit before trying to create a new site. Usually this takes up to 10 seconds.
```sh
# Create ERPNext site
docker run \
-e "SITE_NAME=$SITE_NAME" \
-e "DB_ROOT_USER=$DB_ROOT_USER" \
-e "MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD" \
-e "ADMIN_PASSWORD=$ADMIN_PASSWORD" \
-e "INSTALL_APPS=erpnext" \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
frappe/erpnext-worker:$ERPNEXT_VERSION new
docker-compose exec backend bench new-site <site-name> --mariadb-root-password <db-password> --admin-password <admin-password>
```
#### PostgreSQL Site
If you need to install some app, specify `--install-app`. To see all options, just run `bench new-site --help`.
PostgreSQL is only available v12 onwards. It is NOT available for ERPNext.
It is available as part of `frappe/erpnext-worker`. It inherits from `frappe/frappe-worker`.
To create Postgres site (assuming you already use [Postgres compose override](images-and-compose-files.md#overrides)) you need have to do set `root_login` and `root_password` in common config before that:
```sh
# Create ERPNext site
docker run \
-e "SITE_NAME=$SITE_NAME" \
-e "DB_ROOT_USER=$DB_ROOT_USER" \
-e "POSTGRES_HOST=$POSTGRES_HOST" \
-e "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" \
-e "ADMIN_PASSWORD=$ADMIN_PASSWORD" \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
frappe/erpnext-worker:$ERPNEXT_VERSION new
docker-compose exec backend bench set-config -g root_login <root-login>
docker-compose exec backend bench set-config -g root_password <root-password>
```
Environment Variables needed:
- `SITE_NAME`: name of the new site to create. Site name is domain name that resolves. e.g. `erp.example.com` or `erp.localhost`.
- `DB_ROOT_USER`: MariaDB/PostgreSQL Root user.
- `MYSQL_ROOT_PASSWORD`: In case of the MariaDB docker container use the one set in `MYSQL_ROOT_PASSWORD` in previous steps. In case of a managed database use the appropriate password.
- `MYSQL_ROOT_PASSWORD_FILE` - When the MariaDB root password is stored using docker secrets.
- `ADMIN_PASSWORD`: set the administrator password for the new site.
- `ADMIN_PASSWORD_FILE`: set the administrator password for the new site using docker secrets.
- `INSTALL_APPS=erpnext`: available only in erpnext-worker and erpnext containers (or other containers with custom apps). Installs ERPNext (and/or the specified apps, comma-delinieated) on this new site.
- `FORCE=1`: optional variable which force installation of the same site.
Environment Variables for PostgreSQL only:
- `POSTGRES_HOST`: host for PostgreSQL server
- `POSTGRES_PASSWORD`: Password for `postgres`. The database root user.
Notes:
- To setup existing frappe-bench deployment with default database as PostgreSQL edit the common_site_config.json and set `db_host` to PostgreSQL hostname and `db_port` to PostgreSQL port.
- To create new frappe-bench deployment with default database as PostgreSQL use `POSTGRES_HOST` and `DB_PORT` environment variables in `erpnext-python` service instead of `MARIADB_HOST`
## Add sites to proxy
Change `SITES` variable to the list of sites created encapsulated in backtick and separated by comma with no space. e.g. `` SITES=`site1.example.com`,`site2.example.com` ``.
Reload variables with following command.
Also command is slightly different:
```sh
docker-compose --project-name <project-name> up -d
docker-compose exec backend bench new-site <site-name> --db-type postgres --admin-password <admin-password>
```
## Backup Sites
## Push backup to S3 storage
Environment Variables
- `SITES` is list of sites separated by `:` colon to migrate. e.g. `SITES=site1.domain.com` or `SITES=site1.domain.com:site2.domain.com` By default all sites in bench will be backed up.
- `WITH_FILES` if set to 1, it will backup user-uploaded files.
- By default `backup` takes mariadb dump and gzips it. Example file, `20200325_221230-test_localhost-database.sql.gz`
- If `WITH_FILES` is set then it will also backup public and private files of each site as uncompressed tarball. Example files, `20200325_221230-test_localhost-files.tar` and `20200325_221230-test_localhost-private-files.tar`
- All the files generated by backup are placed at volume location `sites-vol:/{site-name}/private/backups/*`
We have the script that helps to push latest backup to S3.
```sh
docker run \
-e "SITES=site1.domain.com:site2.domain.com" \
-e "WITH_FILES=1" \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
frappe/erpnext-worker:$ERPNEXT_VERSION backup
docker-compose exec backend push_backup.py --site-name <site-name> --bucket <bucket> --region-name <region> --endpoint-url <endpoint-url> --aws-access-key-id <access-key> --aws-secret-access-key <secret-key>
```
The backup will be available in the `sites-vol` volume.
## Push backup to s3 compatible storage
Environment Variables
- `BUCKET_NAME`, Required to set bucket created on S3 compatible storage.
- `REGION`, Required to set region for S3 compatible storage.
- `ACCESS_KEY_ID`, Required to set access key.
- `SECRET_ACCESS_KEY`, Required to set secret access key.
- `ENDPOINT_URL`, Required to set URL of S3 compatible storage.
- `BUCKET_DIR`, Required to set directory in bucket where sites from this deployment will be backed up.
- `BACKUP_LIMIT`, Optionally set this to limit number of backups in bucket directory. Defaults to 3.
```sh
docker run \
-e "BUCKET_NAME=backups" \
-e "REGION=region" \
-e "ACCESS_KEY_ID=access_id_from_provider" \
-e "SECRET_ACCESS_KEY=secret_access_from_provider" \
-e "ENDPOINT_URL=https://region.storage-provider.com" \
-e "BUCKET_DIR=frappe-bench" \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
frappe/frappe-worker:$FRAPPE_VERSION push-backup
```
Note:
- Above example will backup files in bucket called `backup` at location `frappe-bench-v13/site.name.com/DATE_TIME/DATE_TIME-site_name_com-{filetype}.{extension}`,
- example DATE_TIME: 20200325_042020.
- example filetype: database, files or private-files
- example extension: sql.gz or tar
## Restore backups
Environment Variables
- `MYSQL_ROOT_PASSWORD` or `MYSQL_ROOT_PASSWORD_FILE`(when using docker secrets), Required to restore mariadb backups.
- `BUCKET_NAME`, Required to set bucket created on S3 compatible storage.
- `ACCESS_KEY_ID`, Required to set access key.
- `SECRET_ACCESS_KEY`, Required to set secret access key.
- `ENDPOINT_URL`, Required to set URL of S3 compatible storage.
- `REGION`, Required to set region for s3 compatible storage.
- `BUCKET_DIR`, Required to set directory in bucket where sites from this deployment will be backed up.
```sh
docker run \
-e "MYSQL_ROOT_PASSWORD=admin" \
-e "BUCKET_NAME=backups" \
-e "REGION=region" \
-e "ACCESS_KEY_ID=access_id_from_provider" \
-e "SECRET_ACCESS_KEY=secret_access_from_provider" \
-e "ENDPOINT_URL=https://region.storage-provider.com" \
-e "BUCKET_DIR=frappe-bench" \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
-v ./backups:/home/frappe/backups \
--network <project-name>_default \
frappe/frappe-worker:$FRAPPE_VERSION restore-backup
```
Note:
- Volume must be mounted at location `/home/frappe/backups` for restoring sites
- If no backup files are found in volume, it will use s3 credentials to pull backups
- Backup structure for mounted volume or downloaded from s3:
- /home/frappe/backups
- site1.domain.com
- 20200420_162000
- 20200420_162000-site1_domain_com-\*
- site2.domain.com
- 20200420_162000
- 20200420_162000-site2_domain_com-\*
Note that you can restore backup only manually.
## Edit configs
@@ -179,12 +43,11 @@ Editing config manually might be required in some cases,
one such case is to use Amazon RDS (or any other DBaaS).
For full instructions, refer to the [wiki](<https://github.com/frappe/frappe/wiki/Using-Frappe-with-Amazon-RDS-(or-any-other-DBaaS)>). Common question can be found in Issues and on forum.
`common_site_config.json` or `site_config.json` from `sites-vol` volume has to be edited using following command:
`common_site_config.json` or `site_config.json` from `sites` volume has to be edited using following command:
```sh
docker run \
-it \
-v <project-name>_sites-vol:/sites \
docker run --rm -it \
-v <project-name>_sites:/sites \
alpine vi /sites/common_site_config.json
```
@@ -195,8 +58,7 @@ Instead of `alpine` use any image of your choice.
For socketio and gunicorn service ping the hostname:port and that will be sufficient. For workers and scheduler, there is a command that needs to be executed.
```shell
docker exec -it <project-name>_erpnext-worker-d \
docker-entrypoint.sh doctor -p postgresql:5432 --ping-service mongodb:27017
docker-compose exec backend healthcheck.sh --ping-service mongodb:27017
```
Additional services can be pinged as part of health check with option `-p` or `--ping-service`.
@@ -204,86 +66,20 @@ Additional services can be pinged as part of health check with option `-p` or `-
This check ensures that given service should be connected along with services in common_site_config.json.
If connection to service(s) fails, the command fails with exit code 1.
## Frappe internal commands using bench helper
---
To execute commands using bench helper.
```shell
docker run \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
--user frappe \
frappe/frappe-worker:$FRAPPE_VERSION bench --help
```
Example command to clear cache
```shell
docker run \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
--user frappe \
frappe/frappe-worker:$FRAPPE_VERSION bench --site erp.mysite.com clear-cache
```
Notes:
- Use it to install/uninstall custom apps, add system manager user, etc.
- To run the command as non root user add the command option `--user frappe`.
## Delete/Drop Site
#### MariaDB Site
For reference of commands like `backup`, `drop-site` or `migrate` check [official guide](https://frappeframework.com/docs/v13/user/en/bench/frappe-commands) or run:
```sh
# Delete/Drop ERPNext site
docker run \
-e "SITE_NAME=$SITE_NAME" \
-e "DB_ROOT_USER=$DB_ROOT_USER" \
-e "MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD" \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
frappe/erpnext-worker:$ERPNEXT_VERSION drop
docker-compose exec backend bench --help
```
#### PostgreSQL Site
## Migrate site
Note:
- Wait for the `db` service to start and `configurator` to exit before trying to migrate a site. Usually this takes up to 10 seconds.
```sh
# Delete/Drop ERPNext site
docker run \
-e "SITE_NAME=$SITE_NAME" \
-e "DB_ROOT_USER=$DB_ROOT_USER" \
-e "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
frappe/erpnext-worker:$ERPNEXT_VERSION drop
docker-compose exec backend bench --site <site-name> migrate
```
Environment Variables needed:
- `SITE_NAME`: name of the site to be deleted. Site name is domain name that resolves. e.g. `erp.example.com` or `erp.localhost`.
- `DB_ROOT_USER`: MariaDB/PostgreSQL Root user.
- `MYSQL_ROOT_PASSWORD`: Root User password for MariaDB.
- `FORCE=1`: optional variable which force deletion of the same site.
- `NO_BACKUP=1`: option variable to skip the process of taking backup before deleting the site.
Environment Variables for PostgreSQL only:
- `POSTGRES_PASSWORD`: Password for `postgres`. The database root user.
## Migrate Site
```sh
# Migrate ERPNext site
docker run \
-e "MAINTENANCE_MODE=1" \
-v <project-name>_sites-vol:/home/frappe/frappe-bench/sites \
-v <project-name>_assets-vol:/home/frappe/frappe-bench/sites/assets \
--network <project-name>_default \
frappe/erpnext-worker:$ERPNEXT_VERSION migrate
```
Environment Variables needed:
- `MAINTENANCE_MODE`: If set to `1`, this will ensure the bench is switched to maintenance mode during migration.
- `SITES`: Optional list of sites to be migrated, separated by colon (`:`). e.g. `erp.site1.com:erp.site2.com`. If not used, all sites will be migrated by default.