diff --git a/README.md b/README.md index 864b060..29605c6 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,16 @@ - -# Atlassian Data-Center Installation Automation - -## Introduction +# Atlassian Data Center Installation Automation This repository is a suite of Ansible roles, playbooks and support scripts to automate the installation and maintenance of Atlassian Data Center products in cloud environments. -## Usage +On this page: -### Cloud DC-node deployment playbooks +[TOC] + +# Usage + +## Configuring Data Center nodes on cloud deployments The usual scenario for usage as part of a cloud deployment is to invoke the script as part of post-creation actions invoked while a new product node is @@ -23,16 +24,15 @@ In practice, the Ansible roles require some information about the infrastructure that was deployed (e.g. RDS endpoint/password). The way this is currently achieved (on AWS) is that have the CloudFormation template dump this information into the file `/etc/atl` as `RESOURCE_VAR=` lines. This can be then -sourced as environment variables to be retrieved at runtime . See the +sourced as environment variables to be retrieved at runtime. See the helper-script `bin/ansible-with-atl-env` and the corresponding `groups_vars/aws_node_local.yml` var-file. -#### Overriding parameters +## Customizing your deployment -If you want to customise the playbook behaviour the simplest method is to fork -this repository and add your own. However, for some one-off tasks you can also -override the default and calculated settings with special values. To do this, provide -command-line overrides to +To customise playbook behaviour, you can fork this repository and edit it as +needed. However, for one-off tasks you can also override the default and +calculated settings with special values. To do this, provide command-line overrides to [ansible-playbook](https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html). The most likely use-case for this is to download a custom product distribution @@ -61,49 +61,65 @@ them in the `Custom command-line parameters for Ansible` field: -e atl_product_download_url=http://s3.amazon.com/atlassian/jira-9.0.0-PRE-TEST.tar.gz -e atl_use_system_jdk=true -e atl_download_format=tarball +### Other customizable parameters -## Development +For more deployment customization options, consult the following files for parameters you can +override: -### Development philosophy +- [`/roles/product_install/defaults/main.yml`](roles/product_install/defaults/main.yml) +- [`/group_vars/aws_node_local.yml`](group_vars/aws_node_local.yml) -The suite is intended to consist of a number of small, composable roles that can -be combined together into playbooks. Wherever possible the roles should be -platform-agnostic as possible, with platform-specific functionality broken out -into more specific roles. +More customizable parameters are defined in specific roles -- specifically, in the +role's `defaults/main.yml` file. Most of these parameters use the `atl_` prefix. You can +use the following [Bitbucket code search query](https://confluence.atlassian.com/bitbucket/search-873876782.html) +to find them: -Where possible the roles are also product-agnostic (e.g. downloads), with more -specific functionality added in later product-specific roles. + repo:dc-deployments-automation repo:dc-deployments-automation path:*/defaults/main.yml atl -Roles should be reasonably self-contained, with sensible defaults configured in -`/defaults/main.yml` and overridden by the playbook at runtime. Roles may -implicitly depend on variables being defined elsewhere where they cannot define -them natively (e.g. the `jira_config` role depends on the `atl_cluster_node_id` -var being defined; on AWS this is provided by the `aws_common` role, which -should be run first). - -### Development and testing +# Development and testing See [Development](DEVELOPMENT.md) for details on setting up a development environment and running tests. -## Ansible layout +# Roles philosophy + +This suite is intended to consist of many small, composable roles that can +be combined together into playbooks. Wherever possible, roles should be product-agnostic +(e.g. downloads) and platform-agnostic. Functions that are product-specific or +platform-specific are split off into separate roles. + +Roles should be reasonably self-contained, with sensible defaults configured in +`/roles//defaults/main.yml`. Like all playbook parameters, you can override +them at runtime. + +Some roles implicitly depend on other variables beind defined elsewhere. +For example, the `jira_config` role depends on the `atl_cluster_node_id` +var being defined; on AWS this is provided by the `aws_common` role, which +should be run first. + + +# Ansible layout * Helper scripts are in `bin/`. In particular the `bin/ansible-with-atl-env` - wrapper is of use during AWS node initialisation. See _Usage_ above for more - information. + wrapper is of use during AWS node initialisation. Refer to the [Usage](#markdown-header-usage) section for + more information. * Inventory files are under `inv/`. For AWS `cfn-init` the inventory `inv/aws_node_local` inventory is probably what you want. - * Note that this expects the environment to be setup with infrastructure - information; see _Usage_ above. + * Note that this expects the environment to be setup with infrastructure information. + Refer to the [Usage](#markdown-header-usage) section for more information. * Global group vars loaded automatically from `group_vars/.yml`. In particular note `group_vars/aws_node_local.yml` which loads infrastructure information from the environment. -* Roles are under `roles/` - * Platform specific roles start with `_...`, - e.g. `roles/aws_common/`. - * Similarly, product-specific roles should start with `_...`. +* Roles are defined in `roles/` + * Platform specific roles start with `_...`, e.g. `roles/aws_common/`. + * Similarly, product-specific roles should start with `_...`. -## License +# Reporting issues + +If you find any bugs in this repository, or have feature requests or use cases +for us, please raise them in our [public Jira project](https://jira.atlassian.com/projects/SCALE/summary). + +# License Copyright © 2019 Atlassian Corporation Pty Ltd. Licensed under the Apache License, Version 2.0. diff --git a/aws_jira_dc_node.yml b/aws_jira_dc_node.yml index b0470fb..be81d83 100644 --- a/aws_jira_dc_node.yml +++ b/aws_jira_dc_node.yml @@ -7,6 +7,7 @@ atl_product_family: "jira" atl_product_user: "jira" atl_product_edition: "jira-{{ lookup('env', 'ATL_PRODUCT_EDITION') | lower }}" + atl_install_jsd_as_obr: "{{ lookup('env', 'ATL_JSD_ASOBR') }}" atl_systemd_service_name: "jira.service" atl_startup_systemd_params: diff --git a/bitbucket-pipelines.yml b/bitbucket-pipelines.yml index 408b078..b51f0f9 100644 --- a/bitbucket-pipelines.yml +++ b/bitbucket-pipelines.yml @@ -14,7 +14,7 @@ pipelines: - step: name: Pre Parallelization stage script: - - echo "Running tests in 30 batches" + - echo "Running tests in 32 batches" - step: name: Check if number of batches match actual number of scenarios script: @@ -34,7 +34,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 1 - + - step: name: Molecule Test Batch - 2 services: @@ -42,7 +42,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 2 - + - step: name: Molecule Test Batch - 3 services: @@ -50,7 +50,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 3 - + - step: name: Molecule Test Batch - 4 services: @@ -58,7 +58,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 4 - + - step: name: Molecule Test Batch - 5 services: @@ -66,7 +66,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 5 - + - step: name: Molecule Test Batch - 6 services: @@ -74,7 +74,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 6 - + - step: name: Molecule Test Batch - 7 services: @@ -82,7 +82,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 7 - + - step: name: Molecule Test Batch - 8 services: @@ -90,7 +90,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 8 - + - step: name: Molecule Test Batch - 9 services: @@ -98,7 +98,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 9 - + - step: name: Molecule Test Batch - 10 services: @@ -106,7 +106,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 10 - + - step: name: Molecule Test Batch - 11 services: @@ -114,7 +114,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 11 - + - step: name: Molecule Test Batch - 12 services: @@ -122,7 +122,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 12 - + - step: name: Molecule Test Batch - 13 services: @@ -130,7 +130,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 13 - + - step: name: Molecule Test Batch - 14 services: @@ -138,7 +138,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 14 - + - step: name: Molecule Test Batch - 15 services: @@ -146,7 +146,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 15 - + - step: name: Molecule Test Batch - 16 services: @@ -154,7 +154,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 16 - + - step: name: Molecule Test Batch - 17 services: @@ -162,7 +162,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 17 - + - step: name: Molecule Test Batch - 18 services: @@ -170,7 +170,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 18 - + - step: name: Molecule Test Batch - 19 services: @@ -178,7 +178,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 19 - + - step: name: Molecule Test Batch - 20 services: @@ -186,7 +186,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 20 - + - step: name: Molecule Test Batch - 21 services: @@ -194,7 +194,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 21 - + - step: name: Molecule Test Batch - 22 services: @@ -202,7 +202,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 22 - + - step: name: Molecule Test Batch - 23 services: @@ -210,7 +210,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 23 - + - step: name: Molecule Test Batch - 24 services: @@ -218,7 +218,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 24 - + - step: name: Molecule Test Batch - 25 services: @@ -226,7 +226,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 25 - + - step: name: Molecule Test Batch - 26 services: @@ -234,7 +234,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 26 - + - step: name: Molecule Test Batch - 27 services: @@ -242,7 +242,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 27 - + - step: name: Molecule Test Batch - 28 services: @@ -250,7 +250,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 28 - + - step: name: Molecule Test Batch - 29 services: @@ -258,7 +258,7 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 29 - + - step: name: Molecule Test Batch - 30 services: @@ -266,5 +266,21 @@ pipelines: script: - apt-get update && ./bin/install-ansible --dev - ./bin/run-tests-in-batches --batch 30 - - + + - step: + name: Molecule Test Batch - 31 + services: + - docker + script: + - apt-get update && ./bin/install-ansible --dev + - ./bin/run-tests-in-batches --batch 31 + + - step: + name: Molecule Test Batch - 32 + services: + - docker + script: + - apt-get update && ./bin/install-ansible --dev + - ./bin/run-tests-in-batches --batch 32 + + diff --git a/roles/product_install/defaults/main.yml b/roles/product_install/defaults/main.yml index 931e639..6a9940e 100644 --- a/roles/product_install/defaults/main.yml +++ b/roles/product_install/defaults/main.yml @@ -22,12 +22,13 @@ atl_product_download_filename: "{{ atl_download_edition | default(atl_product_ed atl_product_temp_download: "{{ atl_installer_temp }}/{{ atl_product_download_filename }}" atl_product_varfile: "{{ atl_installer_temp }}/{{ atl_product_family }}.varfile" -atl_product_home_shared_download_dir: "{{ atl_product_home_shared }}/downloads" +atl_product_home_shared_download_dir: "{{ atl_shared_mountpoint }}/downloads" atl_product_home_shared_download: "{{ atl_product_home_shared_download_dir }}/{{ atl_product_download_filename }}" atl_product_home_shared_moving_lock: "{{ atl_product_home_shared_download }}_moving" atl_product_home_shared_completed_lock: "{{ atl_product_home_shared_download }}_completed" atl_marketplace_base: "https://marketplace.atlassian.com" +atl_mpac_products: "https://marketplace.atlassian.com/rest/2/products" atl_servicedesk_latest_url: "https://marketplace.atlassian.com/rest/2/products/key/jira-servicedesk/versions/latest" atl_servicedesk_versioned_url: "https://marketplace.atlassian.com/rest/2/products/key/jira-servicedesk/versions/name/{{ atl_product_version }}" atl_servicedesk_url_map: diff --git a/roles/product_install/molecule/bitbucket_latest/tests/test_default.py b/roles/product_install/molecule/bitbucket_latest/tests/test_default.py index bbd851d..0592cbf 100644 --- a/roles/product_install/molecule/bitbucket_latest/tests/test_default.py +++ b/roles/product_install/molecule/bitbucket_latest/tests/test_default.py @@ -37,7 +37,7 @@ def test_latest_is_downloaded(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - installer = host.file('/media/atl/bitbucket/shared/downloads/bitbucket.' + upstream + '-x64.bin') + installer = host.file('/media/atl/downloads/bitbucket.' + upstream + '-x64.bin') assert installer.exists assert installer.user == 'root' @@ -47,6 +47,6 @@ def test_completed_lockfile(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - lockfile = host.file('/media/atl/bitbucket/shared/downloads/bitbucket.' + upstream + '-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/bitbucket.' + upstream + '-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/confluence_latest/tests/test_default.py b/roles/product_install/molecule/confluence_latest/tests/test_default.py index 7ec072b..01044a5 100644 --- a/roles/product_install/molecule/confluence_latest/tests/test_default.py +++ b/roles/product_install/molecule/confluence_latest/tests/test_default.py @@ -35,7 +35,7 @@ def test_latest_is_downloaded(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - installer = host.file('/media/atl/confluence/shared-home/downloads/confluence.'+upstream+'-x64.bin') + installer = host.file('/media/atl/downloads/confluence.'+upstream+'-x64.bin') assert installer.exists assert installer.user == 'root' @@ -44,6 +44,6 @@ def test_completed_lockfile(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - lockfile = host.file('/media/atl/confluence/shared-home/downloads/confluence.'+upstream+'-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/confluence.'+upstream+'-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/crowd_latest/tests/test_default.py b/roles/product_install/molecule/crowd_latest/tests/test_default.py index b75a0b5..6e5dc7c 100644 --- a/roles/product_install/molecule/crowd_latest/tests/test_default.py +++ b/roles/product_install/molecule/crowd_latest/tests/test_default.py @@ -37,7 +37,7 @@ def test_latest_is_downloaded(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - installer = host.file('/media/atl/crowd/shared/downloads/crowd.' + upstream + '.tar.gz') + installer = host.file('/media/atl/downloads/crowd.' + upstream + '.tar.gz') assert installer.exists assert installer.user == 'root' @@ -47,6 +47,6 @@ def test_completed_lockfile(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - lockfile = host.file('/media/atl/crowd/shared/downloads/crowd.' + upstream + '.tar.gz_completed') + lockfile = host.file('/media/atl/downloads/crowd.' + upstream + '.tar.gz_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/default/tests/test_default.py b/roles/product_install/molecule/default/tests/test_default.py index 930ab59..8e20d76 100644 --- a/roles/product_install/molecule/default/tests/test_default.py +++ b/roles/product_install/molecule/default/tests/test_default.py @@ -23,7 +23,7 @@ def test_latest_is_downloaded(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - installer = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'-x64.bin') + installer = host.file('/media/atl/downloads/jira-core.'+upstream+'-x64.bin') assert installer.exists assert installer.user == 'root' @@ -32,6 +32,6 @@ def test_completed_lockfile(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/jira-core.'+upstream+'-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' \ No newline at end of file diff --git a/roles/product_install/molecule/jira_all/Dockerfile.j2 b/roles/product_install/molecule/jira_all/Dockerfile.j2 new file mode 100644 index 0000000..e6aa95d --- /dev/null +++ b/roles/product_install/molecule/jira_all/Dockerfile.j2 @@ -0,0 +1,14 @@ +# Molecule managed + +{% if item.registry is defined %} +FROM {{ item.registry.url }}/{{ item.image }} +{% else %} +FROM {{ item.image }} +{% endif %} + +RUN if [ $(command -v apt-get) ]; then apt-get update && apt-get install -y python sudo bash ca-certificates && apt-get clean; \ + elif [ $(command -v dnf) ]; then dnf makecache && dnf --assumeyes install python sudo python-devel python*-dnf bash && dnf clean all; \ + elif [ $(command -v yum) ]; then yum makecache fast && yum install -y python sudo yum-plugin-ovl bash && sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf && yum clean all; \ + elif [ $(command -v zypper) ]; then zypper refresh && zypper install -y python sudo bash python-xml && zypper clean -a; \ + elif [ $(command -v apk) ]; then apk update && apk add --no-cache python sudo bash ca-certificates; \ + elif [ $(command -v xbps-install) ]; then xbps-install -Syu && xbps-install -y python sudo bash ca-certificates && xbps-remove -O; fi diff --git a/roles/product_install/molecule/jira_all/molecule.yml b/roles/product_install/molecule/jira_all/molecule.yml new file mode 100644 index 0000000..7fd3163 --- /dev/null +++ b/roles/product_install/molecule/jira_all/molecule.yml @@ -0,0 +1,30 @@ +--- +dependency: + name: galaxy +driver: + name: docker +lint: + name: yamllint +platforms: + - name: amazon_linux2 + image: amazonlinux:2 + groups: + - aws_node_local + - name: ubuntu_lts + image: ubuntu:bionic + groups: + - aws_node_local +provisioner: + name: ansible + options: + skip-tags: runtime_pkg + lint: + name: ansible-lint + inventory: + links: + group_vars: ../../../../group_vars/ +verifier: + name: testinfra + lint: + name: flake8 + enabled: false diff --git a/roles/product_install/molecule/jira_all/playbook.yml b/roles/product_install/molecule/jira_all/playbook.yml new file mode 100644 index 0000000..9bddc9a --- /dev/null +++ b/roles/product_install/molecule/jira_all/playbook.yml @@ -0,0 +1,28 @@ +--- +- name: Converge + hosts: all + vars: + atl_product_family: "jira" + atl_product_edition: "jira-software" + atl_product_user: "jira" + atl_product_version: "7.13.1" + atl_install_jsd_as_obr: true + atl_systemd_service_name: "jira.service" + atl_jdbc_encoding: 'UNICODE' + atl_jdbc_collation: 'C' + atl_jdbc_ctype: 'C' + atl_jdbc_template: 'template0' + pre_tasks: + - name: Create cache dir + file: + path: '/media/atl/jira/shared/' + state: directory + - name: Seed version + copy: + dest: '/media/atl/jira/shared/jira-core.version' + content: "7.13.1" + force: false # For idempotency check + roles: + - role: linux_common + - role: product_common + - role: product_install diff --git a/roles/product_install/molecule/jira_all/tests/test_default.py b/roles/product_install/molecule/jira_all/tests/test_default.py new file mode 100644 index 0000000..b9522ec --- /dev/null +++ b/roles/product_install/molecule/jira_all/tests/test_default.py @@ -0,0 +1,48 @@ +import os +from six.moves import urllib + +import testinfra.utils.ansible_runner +import json + +testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner( + os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all') + + +def test_version_is_correct(host): + verfile = host.file('/media/atl/jira/shared/jira-software.version') + assert verfile.exists + + assert verfile.content.decode("UTF-8").strip() == "7.13.1" + +def test_is_downloaded(host): + installer = host.file('/media/atl/downloads/jira-software.7.13.1-x64.bin') + assert installer.exists + assert installer.user == 'root' + +def test_completed_lockfile(host): + lockfile = host.file('/media/atl/downloads/jira-software.7.13.1-x64.bin_completed') + assert lockfile.exists + assert lockfile.user == 'root' + +def test_is_unpacked(host): + installer = host.file('/opt/atlassian/jira-software/7.13.1/atlassian-jira/') + assert installer.exists + assert installer.is_directory + assert installer.user == 'jira' + assert installer.mode == 0o0755 + +def test_obr_is_downloaded(host): + installer = host.file('/media/atl/downloads/jira-servicedesk-application-3.16.1.obr') + assert installer.exists + assert installer.user == 'root' + +def test_obr_completed_lockfile(host): + lockfile = host.file('/media/atl/downloads/jira-servicedesk-application-3.16.1.obr_completed') + assert lockfile.exists + assert lockfile.user == 'root' + +def test_obr_is_unpacked(host): + jsdjar = host.file('/media/atl/jira/shared/plugins/installed-plugins/jira-servicedesk-application-3.16.1.jar') + assert jsdjar.exists + assert jsdjar.user == 'jira' + assert jsdjar.mode == 0o0750 \ No newline at end of file diff --git a/roles/product_install/molecule/jira_cached_with_downgrade/tests/test_default.py b/roles/product_install/molecule/jira_cached_with_downgrade/tests/test_default.py index 788c3de..725ace4 100644 --- a/roles/product_install/molecule/jira_cached_with_downgrade/tests/test_default.py +++ b/roles/product_install/molecule/jira_cached_with_downgrade/tests/test_default.py @@ -14,12 +14,12 @@ def test_version_is_correct(host): assert verfile.content.decode("UTF-8").strip() == "7.10.2" def test_is_downloaded(host): - installer = host.file('/media/atl/jira/shared/downloads/jira-core.7.10.2-x64.bin') + installer = host.file('/media/atl/downloads/jira-core.7.10.2-x64.bin') assert installer.exists assert installer.user == 'root' def test_completed_lockfile(host): - lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.7.10.2-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/jira-core.7.10.2-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/jira_cached_with_upgrade/tests/test_default.py b/roles/product_install/molecule/jira_cached_with_upgrade/tests/test_default.py index 0818e1b..6b215f2 100644 --- a/roles/product_install/molecule/jira_cached_with_upgrade/tests/test_default.py +++ b/roles/product_install/molecule/jira_cached_with_upgrade/tests/test_default.py @@ -14,12 +14,12 @@ def test_version_is_correct(host): assert verfile.content.decode("UTF-8").strip() == "7.10.1" def test_is_downloaded(host): - installer = host.file('/media/atl/jira/shared/downloads/jira-core.7.10.1-x64.bin') + installer = host.file('/media/atl/downloads/jira-core.7.10.1-x64.bin') assert installer.exists assert installer.user == 'root' def test_completed_lockfile(host): - lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.7.10.1-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/jira-core.7.10.1-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/jira_software_latest/tests/test_default.py b/roles/product_install/molecule/jira_software_latest/tests/test_default.py index de1dca3..80d53b8 100644 --- a/roles/product_install/molecule/jira_software_latest/tests/test_default.py +++ b/roles/product_install/molecule/jira_software_latest/tests/test_default.py @@ -35,7 +35,7 @@ def test_latest_is_downloaded(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - installer = host.file('/media/atl/jira/shared/downloads/jira-software.'+upstream+'-x64.bin') + installer = host.file('/media/atl/downloads/jira-software.'+upstream+'-x64.bin') assert installer.exists assert installer.user == 'root' @@ -44,6 +44,6 @@ def test_completed_lockfile(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - lockfile = host.file('/media/atl/jira/shared/downloads/jira-software.'+upstream+'-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/jira-software.'+upstream+'-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' \ No newline at end of file diff --git a/roles/product_install/molecule/jira_tarball/tests/test_default.py b/roles/product_install/molecule/jira_tarball/tests/test_default.py index 11a7438..7155f41 100644 --- a/roles/product_install/molecule/jira_tarball/tests/test_default.py +++ b/roles/product_install/molecule/jira_tarball/tests/test_default.py @@ -23,7 +23,7 @@ def test_latest_is_downloaded(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - installer = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'.tar.gz') + installer = host.file('/media/atl/downloads/jira-core.'+upstream+'.tar.gz') assert installer.exists assert installer.user == 'root' @@ -32,6 +32,6 @@ def test_completed_lockfile(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'.tar.gz_completed') + lockfile = host.file('/media/atl/downloads/jira-core.'+upstream+'.tar.gz_completed') assert lockfile.exists assert lockfile.user == 'root' \ No newline at end of file diff --git a/roles/product_install/molecule/jira_version_from_file/tests/test_default.py b/roles/product_install/molecule/jira_version_from_file/tests/test_default.py index b8a1966..589e7ce 100644 --- a/roles/product_install/molecule/jira_version_from_file/tests/test_default.py +++ b/roles/product_install/molecule/jira_version_from_file/tests/test_default.py @@ -14,12 +14,12 @@ def test_version_is_correct(host): assert verfile.content.decode("UTF-8").strip() == "7.9.0" def test_is_downloaded(host): - installer = host.file('/media/atl/jira/shared/downloads/jira-core.7.9.0-x64.bin') + installer = host.file('/media/atl/downloads/jira-core.7.9.0-x64.bin') assert installer.exists assert installer.user == 'root' def test_completed_lockfile(host): - lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.7.9.0-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/jira-core.7.9.0-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/jira_version_latest/tests/test_default.py b/roles/product_install/molecule/jira_version_latest/tests/test_default.py index 930ab59..8e20d76 100644 --- a/roles/product_install/molecule/jira_version_latest/tests/test_default.py +++ b/roles/product_install/molecule/jira_version_latest/tests/test_default.py @@ -23,7 +23,7 @@ def test_latest_is_downloaded(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - installer = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'-x64.bin') + installer = host.file('/media/atl/downloads/jira-core.'+upstream+'-x64.bin') assert installer.exists assert installer.user == 'root' @@ -32,6 +32,6 @@ def test_completed_lockfile(host): upstream_json = json.load(upstream_fd) upstream = upstream_json['version'] - lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/jira-core.'+upstream+'-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' \ No newline at end of file diff --git a/roles/product_install/molecule/jira_version_override/tests/test_default.py b/roles/product_install/molecule/jira_version_override/tests/test_default.py index 3f16801..e4fa67d 100644 --- a/roles/product_install/molecule/jira_version_override/tests/test_default.py +++ b/roles/product_install/molecule/jira_version_override/tests/test_default.py @@ -14,12 +14,12 @@ def test_version_is_correct(host): assert verfile.content.decode("UTF-8").strip() == "7.13.2" def test_is_downloaded(host): - installer = host.file('/media/atl/jira/shared/downloads/jira-core.7.13.2-x64.bin') + installer = host.file('/media/atl/downloads/jira-core.7.13.2-x64.bin') assert installer.exists assert installer.user == 'root' def test_completed_lockfile(host): - lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.7.13.2-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/jira-core.7.13.2-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/servicedesk3/tests/test_default.py b/roles/product_install/molecule/servicedesk3/tests/test_default.py index 5f50b6e..ae78b3d 100644 --- a/roles/product_install/molecule/servicedesk3/tests/test_default.py +++ b/roles/product_install/molecule/servicedesk3/tests/test_default.py @@ -14,12 +14,12 @@ def test_version_is_correct(host): assert verfile.content.decode("UTF-8").strip() == "3.9.0" def test_is_downloaded(host): - installer = host.file('/media/atl/jira/shared/downloads/servicedesk.3.9.0-x64.bin') + installer = host.file('/media/atl/downloads/servicedesk.3.9.0-x64.bin') assert installer.exists assert installer.user == 'root' def test_completed_lockfile(host): - lockfile = host.file('/media/atl/jira/shared/downloads/servicedesk.3.9.0-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/servicedesk.3.9.0-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/servicedesk4/tests/test_default.py b/roles/product_install/molecule/servicedesk4/tests/test_default.py index 5a22e8c..bc4814e 100644 --- a/roles/product_install/molecule/servicedesk4/tests/test_default.py +++ b/roles/product_install/molecule/servicedesk4/tests/test_default.py @@ -14,12 +14,12 @@ def test_version_is_correct(host): assert verfile.content.decode("UTF-8").strip() == "4.1.0" def test_is_downloaded(host): - installer = host.file('/media/atl/jira/shared/downloads/servicedesk.4.1.0-x64.bin') + installer = host.file('/media/atl/downloads/servicedesk.4.1.0-x64.bin') assert installer.exists assert installer.user == 'root' def test_completed_lockfile(host): - lockfile = host.file('/media/atl/jira/shared/downloads/servicedesk.4.1.0-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/servicedesk.4.1.0-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/molecule/servicedesk_latest/tests/test_default.py b/roles/product_install/molecule/servicedesk_latest/tests/test_default.py index cd975f3..bd4eea8 100644 --- a/roles/product_install/molecule/servicedesk_latest/tests/test_default.py +++ b/roles/product_install/molecule/servicedesk_latest/tests/test_default.py @@ -23,12 +23,12 @@ def test_version_is_correct(host): assert verfile.content.decode("UTF-8").strip() == sd def test_is_downloaded(host): - installer = host.file('/media/atl/jira/shared/downloads/servicedesk.'+sd+'-x64.bin') + installer = host.file('/media/atl/downloads/servicedesk.'+sd+'-x64.bin') assert installer.exists assert installer.user == 'root' def test_completed_lockfile(host): - lockfile = host.file('/media/atl/jira/shared/downloads/servicedesk.'+sd+'-x64.bin_completed') + lockfile = host.file('/media/atl/downloads/servicedesk.'+sd+'-x64.bin_completed') assert lockfile.exists assert lockfile.user == 'root' diff --git a/roles/product_install/tasks/jira-all_extra_tasks.yml b/roles/product_install/tasks/jira-all_extra_tasks.yml new file mode 120000 index 0000000..55832eb --- /dev/null +++ b/roles/product_install/tasks/jira-all_extra_tasks.yml @@ -0,0 +1 @@ +no_op.yml \ No newline at end of file diff --git a/roles/product_install/tasks/jira-servicedesk_as_obr.yml b/roles/product_install/tasks/jira-servicedesk_as_obr.yml new file mode 100644 index 0000000..e602765 --- /dev/null +++ b/roles/product_install/tasks/jira-servicedesk_as_obr.yml @@ -0,0 +1,174 @@ +--- + +- name: Get the installer product version info + uri: + url: "{{ atl_mpac_products }}/key/jira-software/versions/name/{{ atl_product_version }}" + return_content: yes + register: atl_product_version_info + +- name: Show the returned build number + debug: + msg="buildNumber={{ atl_product_version_info.json.buildNumber }}" + +- name: Get the JSD build version info + uri: + url: "{{ atl_mpac_products }}/key/jira-servicedesk/versions/latest?application=\ + jira&applicationBuild={{ atl_product_version_info.json.buildNumber }}" + return_content: yes + register: atl_jsd_build_info + +- name: Show the returned obr binary href + debug: + msg="obr_ref={{ atl_jsd_build_info.json._embedded.artifact._links.binary.href }}" + +- name: how about getting the obr filename + debug: + msg="obr_name=jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr" + +- name: is shared_home set ? + debug: + msg="atl_product_home_shared_download_dir={{ atl_product_home_shared_download_dir }}" +# For the first run a temp obr should be downloaded but moved to +# shared home to ensure all subsequent nodes have access +# to the same specific version binary. +# To prevent a race condition with multiple downloads at the same time +# a directory is used as a lockfile (atomic operation) when moving obr. + +- name: Set assumptions to avoid race condition + set_fact: + download_obr: true + move_obr: false + atl_obr_download_href: "{{ atl_jsd_build_info.json._embedded.artifact._links.binary.href }}" + atl_obr_filename: "jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr" + atl_obr_download: "{{ atl_installer_temp }}/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr" + atl_obr_shared_download: "{{ atl_product_home_shared_download_dir }}/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr" + atl_obr_moving_lock: "{{ atl_product_home_shared_download_dir }}/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr_moving" + atl_obr_completed_lock: "{{ atl_product_home_shared_download_dir }}/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr_completed" + +# Check for pre-downloaded obr on shared_home and completed lock dir. +- name: Check for completed lock directory + stat: + path: "{{ atl_obr_completed_lock }}" + register: completed_lock + +- name: Check for obr in home_shared + stat: + path: "{{ atl_obr_shared_download }}" + register: home_shared_download + +# If obr exists and lockdir exists use this obr instead +- name: Check lock directory and obr exists on shared_home + set_fact: + download_obr: false + atl_obr_download: "{{ atl_obr_shared_download }}" + when: + - home_shared_download.stat.exists + - completed_lock.stat.isdir is defined + - completed_lock.stat.isdir + +# Fetch obr if required +- name: download_obr is true so fetch and do all the things + block: + + # Fetch obr and copy to temp + - name: Fetch obr + get_url: + url: "{{ atl_obr_download_href }}" + dest: "{{ atl_obr_download }}" + mode: 0755 + force: false + register: atl_obr_completed + + # If obr was fetched make the lock directory + - name: Create moving_lock. + file: + path: "{{ atl_obr_moving_lock }}" + state: directory + when: + - atl_obr_completed is succeeded + register: moving_lock_created + + # Directory lock was created by this run? + # If so, then set a fact intending to move obr + - name: Move obr Scenario - lock created by this run + set_fact: + move_obr: true + when: + - moving_lock_created is succeeded + - moving_lock_created.changed + # Otherwise directory lock was either already created or + # could not be created. Fall back is to continue and install from temp + + when: download_obr + +# If the intention is to move obr to home_shared +- name: Move obr to home_shared + block: + + - name: Copy temp installer to home_shared + copy: + src: "{{ atl_obr_download }}" + dest: "{{ atl_obr_shared_download }}" + remote_src: true + register: copied + + - name: Create completed_lock once obr downloaded and copied + file: + path: "{{ atl_obr_completed_lock }}" + state: directory + when: copied is succeeded + register: completed_lock_created + + - name: Remove moving_lock to show that obr is completed + file: + path: "{{ atl_obr_moving_lock }}" + state: absent + when: + - completed_lock_created is succeeded + - copied is succeeded + register: moving_lock_removed + + - name: Delete old temp installer + file: + path: "{{ atl_obr_download }}" + state: absent + when: moving_lock_removed is succeeded + register: temp_deleted + + - name: Set install to home_shared location + set_fact: + atl_obr_download: "{{ atl_obr_shared_download }}" + when: temp_deleted is succeeded + + when: move_obr + +# At this point the binary is in {{ atl_obr_download }} +# (which is either on home_shared or temp) + +- name: Ensure instaled-plugins dir exists + file: + path: "{{ atl_product_home_shared }}/plugins/installed-plugins" + state: directory + mode: 0750 + owner: "{{ atl_product_user }}" + group: "{{ atl_product_user }}" + +# Note as ansible unarchive cant handle "-j junk paths" we need to ignore errors to bypass the path verify +- name: Unpack the obr into the installed-plugins dir + unarchive: + remote_src: true + src: "{{ atl_obr_download }}" + dest: "{{ atl_product_home_shared }}/plugins/installed-plugins" + creates: "{{ atl_product_home_shared }}/plugins/installed-plugins/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.jar" + list_files: no + exclude: + - M* + owner: "{{ atl_product_user }}" + group: "{{ atl_product_user }}" + mode: 0750 + ignore_errors: yes + +- name: Move JSD dependency jars into the installed-plugins dir + shell: + cmd: "mv {{ atl_product_home_shared }}/plugins/installed-plugins/dependencies/*.jar {{ atl_product_home_shared }}/plugins/installed-plugins" + creates: "{{ atl_product_home_shared }}/plugins/installed-plugins/servicedesk-core-ui-plugin-{{ atl_jsd_build_info.json.name }}-*.jar" diff --git a/roles/product_install/tasks/main.yml b/roles/product_install/tasks/main.yml index f136128..81cf0d0 100644 --- a/roles/product_install/tasks/main.yml +++ b/roles/product_install/tasks/main.yml @@ -249,3 +249,9 @@ dest: "{{ atl_product_installation_current }}" state: link force: true + +- name: Include if jsd is requested to be installed from OBR + include_tasks: "jira-servicedesk_as_obr.yml" + when: + - atl_install_jsd_as_obr is defined + - atl_install_jsd_as_obr diff --git a/roles/restore_backups/molecule/restore/Dockerfile.j2 b/roles/restore_backups/molecule/restore/Dockerfile.j2 new file mode 100644 index 0000000..e6aa95d --- /dev/null +++ b/roles/restore_backups/molecule/restore/Dockerfile.j2 @@ -0,0 +1,14 @@ +# Molecule managed + +{% if item.registry is defined %} +FROM {{ item.registry.url }}/{{ item.image }} +{% else %} +FROM {{ item.image }} +{% endif %} + +RUN if [ $(command -v apt-get) ]; then apt-get update && apt-get install -y python sudo bash ca-certificates && apt-get clean; \ + elif [ $(command -v dnf) ]; then dnf makecache && dnf --assumeyes install python sudo python-devel python*-dnf bash && dnf clean all; \ + elif [ $(command -v yum) ]; then yum makecache fast && yum install -y python sudo yum-plugin-ovl bash && sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf && yum clean all; \ + elif [ $(command -v zypper) ]; then zypper refresh && zypper install -y python sudo bash python-xml && zypper clean -a; \ + elif [ $(command -v apk) ]; then apk update && apk add --no-cache python sudo bash ca-certificates; \ + elif [ $(command -v xbps-install) ]; then xbps-install -Syu && xbps-install -y python sudo bash ca-certificates && xbps-remove -O; fi diff --git a/roles/restore_backups/molecule/restore/molecule.yml b/roles/restore_backups/molecule/restore/molecule.yml new file mode 100644 index 0000000..04c0973 --- /dev/null +++ b/roles/restore_backups/molecule/restore/molecule.yml @@ -0,0 +1,30 @@ +--- +dependency: + name: galaxy +driver: + name: docker +lint: + name: yamllint +platforms: + - name: amazon_linux2 + image: amazonlinux:2 + groups: + - aws_node_local + ulimits: + - nofile:262144:262144 +provisioner: + name: ansible + options: + skip-tags: runtime_pkg + lint: + name: ansible-lint + options: + x: ["701"] + inventory: + links: + group_vars: ../../../../group_vars/ +verifier: + name: testinfra + lint: + name: flake8 + enabled: false diff --git a/roles/restore_backups/molecule/restore/playbook.yml b/roles/restore_backups/molecule/restore/playbook.yml new file mode 100644 index 0000000..073d24b --- /dev/null +++ b/roles/restore_backups/molecule/restore/playbook.yml @@ -0,0 +1,75 @@ +--- +- name: Converge + hosts: all + vars: + atl_backup_home_dest: "{{ test_archive }}" + atl_backup_home_restore_canary_path: '/tmp/canary.tmp' + atl_backup_id: 'test-backup' + atl_backup_manifest_url: 'fake_manifest' + + atl_product_edition: 'jira-software' + atl_product_home_shared: '/media/atl/jira/shared' + atl_product_user: 'jira' + atl_product_user_uid: '2001' + atl_product_version_cache: "{{ atl_product_home_shared }}/{{ atl_product_edition }}.version" + + test_archive: '/tmp/hello.tar.gz' + test_archive_file: 'hello.txt' + test_archive_source: '/tmp/hello' + test_pre_step_prefix: '[PRE-TEST]' + test_product_version_file: "/tmp/{{ atl_product_edition }}.version" + + pre_tasks: + - name: "{{ test_pre_step_prefix }} Install tar" + package: + state: present + name: tar + + - name: "{{ test_pre_step_prefix }} Install useradd and groupadd binaries" + package: + state: present + name: shadow-utils + + - name: "{{ test_pre_step_prefix }} Create application group" + group: + name: "{{ atl_product_user }}" + gid: "{{ atl_product_user_uid }}" + + - name: "{{ test_pre_step_prefix }} Create application user" + user: + name: "{{ atl_product_user }}" + uid: "{{ atl_product_user_uid }}" + group: "{{ atl_product_user }}" + + - block: + - name: "{{ test_pre_step_prefix }} Create a directory for the shared home archive" + file: + path: "{{ test_archive_source }}" + state: directory + mode: 0755 + - name: "{{ test_pre_step_prefix }} Create a file in the shared home" + lineinfile: + create: yes + line: 'Hello, world!' + path: "{{ test_archive_source }}/{{ test_archive_file }}" + mode: 0640 + - name: "{{ test_pre_step_prefix }} Create the version file in the shared home" + lineinfile: + create: yes + line: '8.5' + path: "{{ test_product_version_file }}" + mode: 0640 + - name: "{{ test_pre_step_prefix }} Archive the shared home" + archive: + path: + - "{{ test_archive_source }}" + - "{{ test_product_version_file }}" + dest: "{{ test_archive }}" + owner: "{{ atl_product_user }}" + + tasks: + - name: Install distro-specific restore support packages + include_tasks: "../../tasks/{{ ansible_distribution|lower }}.yml" + + - name: Restore shared home + include_tasks: "../../tasks/home_restore.yml" diff --git a/roles/restore_backups/molecule/restore/tests/test_default.py b/roles/restore_backups/molecule/restore/tests/test_default.py new file mode 100644 index 0000000..bcfa53d --- /dev/null +++ b/roles/restore_backups/molecule/restore/tests/test_default.py @@ -0,0 +1,39 @@ +import os +import pytest + +import testinfra.utils.ansible_runner + +testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner( + os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all') + + +@pytest.mark.parametrize('exe', [ + '/usr/bin/pg_dump', + '/usr/bin/pg_restore', + '/usr/bin/psql' +]) +def test_postgresql_amazon_linux_extras_exes(host, exe): + assert host.file(exe).exists + +def test_postgresql_version(host): + pg_dump_version_output = host.check_output('pg_dump --version') + assert '(PostgreSQL) 9.6' in pg_dump_version_output + +@pytest.mark.parametrize('file', [ + '/media/atl/jira/shared', + '/media/atl/jira/shared/hello', + '/media/atl/jira/shared/hello/hello.txt' +]) +def test_shared_home_owner(host, file): + assert host.file(file).exists + assert host.file(file).user == 'jira' + assert host.file(file).group == 'jira' + +def test_file_modes(host): + assert host.file('/media/atl/jira/shared/hello').mode == 0o755 + assert host.file('/media/atl/jira/shared/hello/hello.txt').mode == 0o640 + +def test_version_file_owned_by_root(host): + assert host.file('/media/atl/jira/shared/jira-software.version').exists + assert host.file('/media/atl/jira/shared/jira-software.version').user == 'root' + assert host.file('/media/atl/jira/shared/jira-software.version').group == 'root' \ No newline at end of file diff --git a/roles/restore_backups/tasks/amazon.yml b/roles/restore_backups/tasks/amazon.yml index 9a61d16..f18a410 100644 --- a/roles/restore_backups/tasks/amazon.yml +++ b/roles/restore_backups/tasks/amazon.yml @@ -8,3 +8,5 @@ PYTHON: /bin/python args: creates: /usr/bin/psql + environment: + PYTHON: /bin/python diff --git a/roles/restore_backups/tasks/home_restore.yml b/roles/restore_backups/tasks/home_restore.yml new file mode 100644 index 0000000..4c5f2ad --- /dev/null +++ b/roles/restore_backups/tasks/home_restore.yml @@ -0,0 +1,44 @@ +--- +- name: Check for the restore canary file + stat: + path: "{{ atl_backup_home_restore_canary_path }}" + register: restore_canary + +- block: + - name: Create shared home if necessary + file: + path: "{{ atl_product_home_shared }}" + state: directory + mode: 0750 + owner: "{{ atl_product_user }}" + group: "{{ atl_product_user }}" + + - name: Restore the shared-home backup + unarchive: + src: "{{ atl_backup_home_dest }}" + remote_src: yes + dest: "{{ atl_product_home_shared }}" + + - name: Set shared home owner and group to application user + file: + path: "{{ atl_product_home_shared }}" + recurse: yes + group: "{{ atl_product_user }}" + owner: "{{ atl_product_user }}" + state: directory + + - name: Set version file owner and group to root + file: + path: "{{ atl_product_version_cache }}" + group: root + owner: root + state: file + # Ignore the error in case there is no product version file in the backup + ignore_errors: yes + + - name: Create restore-canary if necessary + copy: + dest: "{{ atl_backup_home_restore_canary_path }}" + content: "{{ atl_backup_id }}" + + when: not restore_canary.stat.exists diff --git a/roles/restore_backups/tasks/main.yml b/roles/restore_backups/tasks/main.yml index 6b28bb1..1c81430 100644 --- a/roles/restore_backups/tasks/main.yml +++ b/roles/restore_backups/tasks/main.yml @@ -68,7 +68,9 @@ overwrite: different bucket: "{{ atl_backup_manifest.artifacts.db.location.location | urlsplit('hostname') }}" object: "{{ atl_backup_manifest.artifacts.db.location.location | urlsplit('path') }}" - dest: "{{ atl_backup_db_dest }}" + # We save the backup as a .tar file so that the postgresql_db module uses pg_restore instead of psql to do restore + # This can be removed when ansible 2.10 is released + dest: "{{ atl_backup_db_dest }}.tar" - name: Fetch Home backup from S3 aws_s3: @@ -96,42 +98,18 @@ template: "{{ atl_jdbc_template }}" # Depends on fetch_backup roles state: restore - target: "{{ atl_backup_db_dest }}" + target: "{{ atl_backup_db_dest }}.tar" + target_opts: "-Fc" register: result + # managed DBs in cloud providers are not allowing full root access to the DB engine, we can safely ignore the COMMENT ON EXTENSION error failed_when: - result.rc != 0 - - "COMMENT ON EXTENSION" not in result.msg + - '"COMMENT ON EXTENSION" not in result.msg' when: db_created.changed and atl_backup_db_dest is defined - - name: Check for the restore canary file - stat: - path: "{{ atl_backup_home_restore_canary_path }}" - register: restore_canary - - - block: - - - name: Create shared home if necessary - file: - path: "{{ atl_product_home_shared }}" - state: directory - mode: 0750 - owner: "{{ atl_product_user }}" - group: "{{ atl_product_user }}" - - - name: Restore the shared-home backup - unarchive: - src: "{{ atl_backup_home_dest }}" - dest: "{{ atl_product_home_shared }}" - owner: "{{ atl_product_user }}" - group: "{{ atl_product_user }}" - - - name: Create restore-canary if necessary - copy: - dest: "{{ atl_backup_home_restore_canary_path }}" - content: "{{ atl_backup_id }}" - - when: not restore_canary.stat.exists + - name: Restore shared home + include_tasks: "home_restore.yml" when: atl_restore_required