From ebfee83f9fc25c964905acf53f643bd4c83b522b Mon Sep 17 00:00:00 2001 From: Lev Vereshchagin Date: Fri, 17 Dec 2021 15:06:32 +0300 Subject: [PATCH] Update site-operations docs --- docs/site-operations.md | 256 ++++------------------------------------ 1 file changed, 20 insertions(+), 236 deletions(-) diff --git a/docs/site-operations.md b/docs/site-operations.md index bf7ac0c3..cc73036c 100644 --- a/docs/site-operations.md +++ b/docs/site-operations.md @@ -1,177 +1,39 @@ # Site operations -Create and use env file to pass environment variables to containers, - -```sh -source .env -``` - -Or specify environment variables instead of passing secrets as command arguments. Refer notes section for environment variables required - -## Setup New Site +## Setup new site Note: -- Wait for the database service to start before trying to create a new site. - - If new site creation fails, retry after the MariaDB container is up and running. - - If you're using a managed database instance, make sure that the database is running before setting up a new site. - -#### MariaDB Site +- Wait for the `db` service to start and `configurator` to exit before trying to create a new site. Usually this takes up to 10 seconds. ```sh -# Create ERPNext site -docker run \ - -e "SITE_NAME=$SITE_NAME" \ - -e "DB_ROOT_USER=$DB_ROOT_USER" \ - -e "MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD" \ - -e "ADMIN_PASSWORD=$ADMIN_PASSWORD" \ - -e "INSTALL_APPS=erpnext" \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - --network _default \ - frappe/erpnext-worker:$VERSION new +docker-compose exec backend bench new-site --mariadb-root-password --admin-password ``` -#### PostgreSQL Site +If you need to install some app, specify `--install-app`. To see all options, just run `bench new-site --help`. -PostgreSQL is only available v12 onwards. It is NOT available for ERPNext. -It is available as part of `frappe/erpnext-worker`. It inherits from `frappe/frappe-worker`. +To create Postgres site (assuming you already use [Postgres compose override](images-and-compose-files.md#overrides)) you need have to do set `root_login` and `root_password` in common config before that: ```sh -# Create ERPNext site -docker run \ - -e "SITE_NAME=$SITE_NAME" \ - -e "DB_ROOT_USER=$DB_ROOT_USER" \ - -e "POSTGRES_HOST=$POSTGRES_HOST" \ - -e "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" \ - -e "ADMIN_PASSWORD=$ADMIN_PASSWORD" \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - --network _default \ - frappe/erpnext-worker:$VERSION new +docker-compose exec backend bench set-config -g root_login +docker-compose exec backend bench set-config -g root_password ``` -Environment Variables needed: - -- `SITE_NAME`: name of the new site to create. Site name is domain name that resolves. e.g. `erp.example.com` or `erp.localhost`. -- `DB_ROOT_USER`: MariaDB/PostgreSQL Root user. -- `MYSQL_ROOT_PASSWORD`: In case of the MariaDB docker container use the one set in `MYSQL_ROOT_PASSWORD` in previous steps. In case of a managed database use the appropriate password. -- `MYSQL_ROOT_PASSWORD_FILE` - When the MariaDB root password is stored using docker secrets. -- `ADMIN_PASSWORD`: set the administrator password for the new site. -- `ADMIN_PASSWORD_FILE`: set the administrator password for the new site using docker secrets. -- `INSTALL_APPS=erpnext`: available only in erpnext-worker and erpnext containers (or other containers with custom apps). Installs ERPNext (and/or the specified apps, comma-delinieated) on this new site. -- `FORCE=1`: optional variable which force installation of the same site. - -Environment Variables for PostgreSQL only: - -- `POSTGRES_HOST`: host for PostgreSQL server -- `POSTGRES_PASSWORD`: Password for `postgres`. The database root user. - -Notes: - -- To setup existing frappe-bench deployment with default database as PostgreSQL edit the common_site_config.json and set `db_host` to PostgreSQL hostname and `db_port` to PostgreSQL port. -- To create new frappe-bench deployment with default database as PostgreSQL use `POSTGRES_HOST` and `DB_PORT` environment variables in `erpnext-python` service instead of `MARIADB_HOST` - -## Add sites to proxy - -Change `SITES` variable to the list of sites created encapsulated in backtick and separated by comma with no space. e.g. `` SITES=`site1.example.com`,`site2.example.com` ``. - -Reload variables with following command. +Also command is slightly different: ```sh -docker-compose --project-name up -d +docker-compose exec backend bench new-site --db-type postgres --admin-password ``` -## Backup Sites +## Push backup to S3 storage -Environment Variables - -- `SITES` is list of sites separated by `:` colon to migrate. e.g. `SITES=site1.domain.com` or `SITES=site1.domain.com:site2.domain.com` By default all sites in bench will be backed up. -- `WITH_FILES` if set to 1, it will backup user-uploaded files. -- By default `backup` takes mariadb dump and gzips it. Example file, `20200325_221230-test_localhost-database.sql.gz` -- If `WITH_FILES` is set then it will also backup public and private files of each site as uncompressed tarball. Example files, `20200325_221230-test_localhost-files.tar` and `20200325_221230-test_localhost-private-files.tar` -- All the files generated by backup are placed at volume location `sites-vol:/{site-name}/private/backups/*` +We have the script that helps to push latest backup to S3. ```sh -docker run \ - -e "SITES=site1.domain.com:site2.domain.com" \ - -e "WITH_FILES=1" \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - --network _default \ - frappe/erpnext-worker:$VERSION backup +docker-compose exec backend push_backup.py --site-name --bucket --region-name --endpoint-url --aws-access-key-id --aws-secret-access-key ``` -The backup will be available in the `sites-vol` volume. - -## Push backup to s3 compatible storage - -Environment Variables - -- `BUCKET_NAME`, Required to set bucket created on S3 compatible storage. -- `REGION`, Required to set region for S3 compatible storage. -- `ACCESS_KEY_ID`, Required to set access key. -- `SECRET_ACCESS_KEY`, Required to set secret access key. -- `ENDPOINT_URL`, Required to set URL of S3 compatible storage. -- `BUCKET_DIR`, Required to set directory in bucket where sites from this deployment will be backed up. -- `BACKUP_LIMIT`, Optionally set this to limit number of backups in bucket directory. Defaults to 3. - -```sh - docker run \ - -e "BUCKET_NAME=backups" \ - -e "REGION=region" \ - -e "ACCESS_KEY_ID=access_id_from_provider" \ - -e "SECRET_ACCESS_KEY=secret_access_from_provider" \ - -e "ENDPOINT_URL=https://region.storage-provider.com" \ - -e "BUCKET_DIR=frappe-bench" \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - --network _default \ - frappe/frappe-worker:$VERSION push-backup -``` - -Note: - -- Above example will backup files in bucket called `backup` at location `frappe-bench-v13/site.name.com/DATE_TIME/DATE_TIME-site_name_com-{filetype}.{extension}`, -- example DATE_TIME: 20200325_042020. -- example filetype: database, files or private-files -- example extension: sql.gz or tar - -## Restore backups - -Environment Variables - -- `MYSQL_ROOT_PASSWORD` or `MYSQL_ROOT_PASSWORD_FILE`(when using docker secrets), Required to restore mariadb backups. -- `BUCKET_NAME`, Required to set bucket created on S3 compatible storage. -- `ACCESS_KEY_ID`, Required to set access key. -- `SECRET_ACCESS_KEY`, Required to set secret access key. -- `ENDPOINT_URL`, Required to set URL of S3 compatible storage. -- `REGION`, Required to set region for s3 compatible storage. -- `BUCKET_DIR`, Required to set directory in bucket where sites from this deployment will be backed up. - -```sh -docker run \ - -e "MYSQL_ROOT_PASSWORD=admin" \ - -e "BUCKET_NAME=backups" \ - -e "REGION=region" \ - -e "ACCESS_KEY_ID=access_id_from_provider" \ - -e "SECRET_ACCESS_KEY=secret_access_from_provider" \ - -e "ENDPOINT_URL=https://region.storage-provider.com" \ - -e "BUCKET_DIR=frappe-bench" \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - -v ./backups:/home/frappe/backups \ - --network _default \ - frappe/frappe-worker:$VERSION restore-backup -``` - -Note: - -- Volume must be mounted at location `/home/frappe/backups` for restoring sites -- If no backup files are found in volume, it will use s3 credentials to pull backups -- Backup structure for mounted volume or downloaded from s3: - - /home/frappe/backups - - site1.domain.com - - 20200420_162000 - - 20200420_162000-site1_domain_com-\* - - site2.domain.com - - 20200420_162000 - - 20200420_162000-site2_domain_com-\* +Note that you can restore backup only manually. ## Edit configs @@ -179,12 +41,11 @@ Editing config manually might be required in some cases, one such case is to use Amazon RDS (or any other DBaaS). For full instructions, refer to the [wiki](). Common question can be found in Issues and on forum. -`common_site_config.json` or `site_config.json` from `sites-vol` volume has to be edited using following command: +`common_site_config.json` or `site_config.json` from `sites` volume has to be edited using following command: ```sh -docker run \ - -it \ - -v _sites-vol:/sites \ +docker run --rm -it \ + -v _sites:/sites \ alpine vi /sites/common_site_config.json ``` @@ -195,8 +56,7 @@ Instead of `alpine` use any image of your choice. For socketio and gunicorn service ping the hostname:port and that will be sufficient. For workers and scheduler, there is a command that needs to be executed. ```shell -docker exec -it _erpnext-worker-d \ -docker-entrypoint.sh doctor -p postgresql:5432 --ping-service mongodb:27017 +docker-compose exec backend healthcheck.sh --ping-service mongodb:27017 ``` Additional services can be pinged as part of health check with option `-p` or `--ping-service`. @@ -204,86 +64,10 @@ Additional services can be pinged as part of health check with option `-p` or `- This check ensures that given service should be connected along with services in common_site_config.json. If connection to service(s) fails, the command fails with exit code 1. -## Frappe internal commands using bench helper +--- -To execute commands using bench helper. - -```shell - docker run \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - --network _default \ - --user frappe \ - frappe/frappe-worker:$VERSION bench --help -``` - -Example command to clear cache - -```shell - docker run \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - --network _default \ - --user frappe \ - frappe/frappe-worker:$VERSION bench --site erp.mysite.com clear-cache -``` - -Notes: - -- Use it to install/uninstall custom apps, add system manager user, etc. -- To run the command as non root user add the command option `--user frappe`. - -## Delete/Drop Site - -#### MariaDB Site +For reference of commands like `backup`, `drop-site` or `migrate` check [official guide](https://frappeframework.com/docs/v13/user/en/bench/frappe-commands) or run: ```sh -# Delete/Drop ERPNext site -docker run \ - -e "SITE_NAME=$SITE_NAME" \ - -e "DB_ROOT_USER=$DB_ROOT_USER" \ - -e "MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD" \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - --network _default \ - frappe/erpnext-worker:$VERSION drop +docker-compose exec backend bench --help ``` - -#### PostgreSQL Site - -```sh -# Delete/Drop ERPNext site -docker run \ - -e "SITE_NAME=$SITE_NAME" \ - -e "DB_ROOT_USER=$DB_ROOT_USER" \ - -e "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - --network _default \ - frappe/erpnext-worker:$VERSION drop -``` - -Environment Variables needed: - -- `SITE_NAME`: name of the site to be deleted. Site name is domain name that resolves. e.g. `erp.example.com` or `erp.localhost`. -- `DB_ROOT_USER`: MariaDB/PostgreSQL Root user. -- `MYSQL_ROOT_PASSWORD`: Root User password for MariaDB. -- `FORCE=1`: optional variable which force deletion of the same site. -- `NO_BACKUP=1`: option variable to skip the process of taking backup before deleting the site. - -Environment Variables for PostgreSQL only: - -- `POSTGRES_PASSWORD`: Password for `postgres`. The database root user. - -## Migrate Site - -```sh -# Migrate ERPNext site -docker run \ - -e "MAINTENANCE_MODE=1" \ - -v _sites-vol:/home/frappe/frappe-bench/sites \ - -v _assets-vol:/home/frappe/frappe-bench/sites/assets \ - --network _default \ - frappe/erpnext-worker:$ERPNEXT_VERSION migrate -``` - -Environment Variables needed: - -- `MAINTENANCE_MODE`: If set to `1`, this will ensure the bench is switched to maintenance mode during migration. -- `SITES`: Optional list of sites to be migrated, separated by colon (`:`). e.g. `erp.site1.com:erp.site2.com`. If not used, all sites will be migrated by default.