blsmith/dc-deployments-automation
1
0
Fork 0
mirror of https://bitbucket.org/atlassian/dc-deployments-automation.git synced 2025-12-15 09:23:06 -06:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into DCD-798-fix-for-pg_dump-cusstom-format

Browse Source
This commit is contained in:
Varun Arbatti
2019-11-22 15:58:34 +11:00
parent 004d774d16 b64355b8ca
commit 2032593246
32 changed files with 647 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files

92
README.md
View File

@@ -1,15 +1,16 @@
# Atlassian Data-Center Installation Automation
## Introduction
# Atlassian Data Center Installation Automation
This repository is a suite of Ansible roles, playbooks and support scripts to
automate the installation and maintenance of Atlassian Data Center products in
cloud environments.
## Usage
On this page:
### Cloud DC-node deployment playbooks
[TOC]
# Usage
## Configuring Data Center nodes on cloud deployments
The usual scenario for usage as part of a cloud deployment is to invoke the
script as part of post-creation actions invoked while a new product node is
@@ -23,16 +24,15 @@ In practice, the Ansible roles require some information about the infrastructure
that was deployed (e.g. RDS endpoint/password). The way this is currently
achieved (on AWS) is that have the CloudFormation template dump this information
into the file `/etc/atl` as `RESOURCE_VAR=<resource>` lines. This can be then
sourced as environment variables to be retrieved at runtime . See the
sourced as environment variables to be retrieved at runtime. See the
helper-script `bin/ansible-with-atl-env` and the corresponding
`groups_vars/aws_node_local.yml` var-file.
#### Overriding parameters
## Customizing your deployment
If you want to customise the playbook behaviour the simplest method is to fork
this repository and add your own. However, for some one-off tasks you can also
override the default and calculated settings with special values. To do this, provide
command-line overrides to
To customise playbook behaviour, you can fork this repository and edit it as
needed. However, for one-off tasks you can also override the default and
calculated settings with special values. To do this, provide command-line overrides to
[ansible-playbook](https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html).
The most likely use-case for this is to download a custom product distribution
@@ -61,49 +61,65 @@ them in the `Custom command-line parameters for Ansible` field:
-e atl_product_download_url=http://s3.amazon.com/atlassian/jira-9.0.0-PRE-TEST.tar.gz -e atl_use_system_jdk=true -e atl_download_format=tarball
### Other customizable parameters
## Development
For more deployment customization options, consult the following files for parameters you can
override:
### Development philosophy
- [`/roles/product_install/defaults/main.yml`](roles/product_install/defaults/main.yml)
- [`/group_vars/aws_node_local.yml`](group_vars/aws_node_local.yml)
The suite is intended to consist of a number of small, composable roles that can
be combined together into playbooks. Wherever possible the roles should be
platform-agnostic as possible, with platform-specific functionality broken out
into more specific roles.
More customizable parameters are defined in specific roles -- specifically, in the
role's `defaults/main.yml` file. Most of these parameters use the `atl_` prefix. You can
use the following [Bitbucket code search query](https://confluence.atlassian.com/bitbucket/search-873876782.html)
to find them:
Where possible the roles are also product-agnostic (e.g. downloads), with more
specific functionality added in later product-specific roles.
repo:dc-deployments-automation repo:dc-deployments-automation path:*/defaults/main.yml atl
Roles should be reasonably self-contained, with sensible defaults configured in
`<role>/defaults/main.yml` and overridden by the playbook at runtime. Roles may
implicitly depend on variables being defined elsewhere where they cannot define
them natively (e.g. the `jira_config` role depends on the `atl_cluster_node_id`
var being defined; on AWS this is provided by the `aws_common` role, which
should be run first).
### Development and testing
# Development and testing
See [Development](DEVELOPMENT.md) for details on setting up a development
environment and running tests.
## Ansible layout
# Roles philosophy
This suite is intended to consist of many small, composable roles that can
be combined together into playbooks. Wherever possible, roles should be product-agnostic
(e.g. downloads) and platform-agnostic. Functions that are product-specific or
platform-specific are split off into separate roles.
Roles should be reasonably self-contained, with sensible defaults configured in
`/roles/<role>/defaults/main.yml`. Like all playbook parameters, you can override
them at runtime.
Some roles implicitly depend on other variables beind defined elsewhere.
For example, the `jira_config` role depends on the `atl_cluster_node_id`
var being defined; on AWS this is provided by the `aws_common` role, which
should be run first.
# Ansible layout
* Helper scripts are in `bin/`. In particular the `bin/ansible-with-atl-env`
wrapper is of use during AWS node initialisation. See _Usage_ above for more
information.
wrapper is of use during AWS node initialisation. Refer to the [Usage](#markdown-header-usage) section for
more information.
* Inventory files are under `inv/`. For AWS `cfn-init` the inventory
`inv/aws_node_local` inventory is probably what you want.
* Note that this expects the environment to be setup with infrastructure
information; see _Usage_ above.
* Note that this expects the environment to be setup with infrastructure information.
Refer to the [Usage](#markdown-header-usage) section for more information.
* Global group vars loaded automatically from `group_vars/<group>.yml`. In
particular note `group_vars/aws_node_local.yml` which loads infrastructure
information from the environment.
* Roles are under `roles/`
* Platform specific roles start with `<platform-shortname>_...`,
e.g. `roles/aws_common/`.
* Similarly, product-specific roles should start with `<product>_...`.
* Roles are defined in `roles/`
* Platform specific roles start with `<platform-shortname>_...`, e.g. `roles/aws_common/`.
* Similarly, product-specific roles should start with `<product>_...`.
## License
# Reporting issues
If you find any bugs in this repository, or have feature requests or use cases
for us, please raise them in our [public Jira project](https://jira.atlassian.com/projects/SCALE/summary).
# License
Copyright © 2019 Atlassian Corporation Pty Ltd.
Licensed under the Apache License, Version 2.0.

1
aws_jira_dc_node.yml
View File

@@ -7,6 +7,7 @@
atl_product_family: "jira"
atl_product_user: "jira"
atl_product_edition: "jira-{{ lookup('env', 'ATL_PRODUCT_EDITION') | lower }}"
atl_install_jsd_as_obr: "{{ lookup('env', 'ATL_JSD_ASOBR') }}"
atl_systemd_service_name: "jira.service"
atl_startup_systemd_params:

80
bitbucket-pipelines.yml
View File

@@ -14,7 +14,7 @@ pipelines:
- step:
name: Pre Parallelization stage
script:
- echo "Running tests in 30 batches"
- echo "Running tests in 32 batches"
- step:
name: Check if number of batches match actual number of scenarios
script:
@@ -34,7 +34,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 1
- step:
name: Molecule Test Batch - 2
services:
@@ -42,7 +42,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 2
- step:
name: Molecule Test Batch - 3
services:
@@ -50,7 +50,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 3
- step:
name: Molecule Test Batch - 4
services:
@@ -58,7 +58,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 4
- step:
name: Molecule Test Batch - 5
services:
@@ -66,7 +66,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 5
- step:
name: Molecule Test Batch - 6
services:
@@ -74,7 +74,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 6
- step:
name: Molecule Test Batch - 7
services:
@@ -82,7 +82,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 7
- step:
name: Molecule Test Batch - 8
services:
@@ -90,7 +90,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 8
- step:
name: Molecule Test Batch - 9
services:
@@ -98,7 +98,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 9
- step:
name: Molecule Test Batch - 10
services:
@@ -106,7 +106,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 10
- step:
name: Molecule Test Batch - 11
services:
@@ -114,7 +114,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 11
- step:
name: Molecule Test Batch - 12
services:
@@ -122,7 +122,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 12
- step:
name: Molecule Test Batch - 13
services:
@@ -130,7 +130,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 13
- step:
name: Molecule Test Batch - 14
services:
@@ -138,7 +138,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 14
- step:
name: Molecule Test Batch - 15
services:
@@ -146,7 +146,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 15
- step:
name: Molecule Test Batch - 16
services:
@@ -154,7 +154,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 16
- step:
name: Molecule Test Batch - 17
services:
@@ -162,7 +162,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 17
- step:
name: Molecule Test Batch - 18
services:
@@ -170,7 +170,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 18
- step:
name: Molecule Test Batch - 19
services:
@@ -178,7 +178,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 19
- step:
name: Molecule Test Batch - 20
services:
@@ -186,7 +186,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 20
- step:
name: Molecule Test Batch - 21
services:
@@ -194,7 +194,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 21
- step:
name: Molecule Test Batch - 22
services:
@@ -202,7 +202,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 22
- step:
name: Molecule Test Batch - 23
services:
@@ -210,7 +210,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 23
- step:
name: Molecule Test Batch - 24
services:
@@ -218,7 +218,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 24
- step:
name: Molecule Test Batch - 25
services:
@@ -226,7 +226,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 25
- step:
name: Molecule Test Batch - 26
services:
@@ -234,7 +234,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 26
- step:
name: Molecule Test Batch - 27
services:
@@ -242,7 +242,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 27
- step:
name: Molecule Test Batch - 28
services:
@@ -250,7 +250,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 28
- step:
name: Molecule Test Batch - 29
services:
@@ -258,7 +258,7 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 29
- step:
name: Molecule Test Batch - 30
services:
@@ -266,5 +266,21 @@ pipelines:
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 30
- step:
name: Molecule Test Batch - 31
services:
- docker
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 31
- step:
name: Molecule Test Batch - 32
services:
- docker
script:
- apt-get update && ./bin/install-ansible --dev
- ./bin/run-tests-in-batches --batch 32

3
roles/product_install/defaults/main.yml
View File

@@ -22,12 +22,13 @@ atl_product_download_filename: "{{ atl_download_edition | default(atl_product_ed
atl_product_temp_download: "{{ atl_installer_temp }}/{{ atl_product_download_filename }}"
atl_product_varfile: "{{ atl_installer_temp }}/{{ atl_product_family }}.varfile"
atl_product_home_shared_download_dir: "{{ atl_product_home_shared }}/downloads"
atl_product_home_shared_download_dir: "{{ atl_shared_mountpoint }}/downloads"
atl_product_home_shared_download: "{{ atl_product_home_shared_download_dir }}/{{ atl_product_download_filename }}"
atl_product_home_shared_moving_lock: "{{ atl_product_home_shared_download }}_moving"
atl_product_home_shared_completed_lock: "{{ atl_product_home_shared_download }}_completed"
atl_marketplace_base: "https://marketplace.atlassian.com"
atl_mpac_products: "https://marketplace.atlassian.com/rest/2/products"
atl_servicedesk_latest_url: "https://marketplace.atlassian.com/rest/2/products/key/jira-servicedesk/versions/latest"
atl_servicedesk_versioned_url: "https://marketplace.atlassian.com/rest/2/products/key/jira-servicedesk/versions/name/{{ atl_product_version }}"
atl_servicedesk_url_map:

4
roles/product_install/molecule/bitbucket_latest/tests/test_default.py
View File

@@ -37,7 +37,7 @@ def test_latest_is_downloaded(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
installer = host.file('/media/atl/bitbucket/shared/downloads/bitbucket.' + upstream + '-x64.bin')
installer = host.file('/media/atl/downloads/bitbucket.' + upstream + '-x64.bin')
assert installer.exists
assert installer.user == 'root'
@@ -47,6 +47,6 @@ def test_completed_lockfile(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
lockfile = host.file('/media/atl/bitbucket/shared/downloads/bitbucket.' + upstream + '-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/bitbucket.' + upstream + '-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/confluence_latest/tests/test_default.py
View File

@@ -35,7 +35,7 @@ def test_latest_is_downloaded(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
installer = host.file('/media/atl/confluence/shared-home/downloads/confluence.'+upstream+'-x64.bin')
installer = host.file('/media/atl/downloads/confluence.'+upstream+'-x64.bin')
assert installer.exists
assert installer.user == 'root'
@@ -44,6 +44,6 @@ def test_completed_lockfile(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
lockfile = host.file('/media/atl/confluence/shared-home/downloads/confluence.'+upstream+'-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/confluence.'+upstream+'-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/crowd_latest/tests/test_default.py
View File

@@ -37,7 +37,7 @@ def test_latest_is_downloaded(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
installer = host.file('/media/atl/crowd/shared/downloads/crowd.' + upstream + '.tar.gz')
installer = host.file('/media/atl/downloads/crowd.' + upstream + '.tar.gz')
assert installer.exists
assert installer.user == 'root'
@@ -47,6 +47,6 @@ def test_completed_lockfile(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
lockfile = host.file('/media/atl/crowd/shared/downloads/crowd.' + upstream + '.tar.gz_completed')
lockfile = host.file('/media/atl/downloads/crowd.' + upstream + '.tar.gz_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/default/tests/test_default.py
View File

@@ -23,7 +23,7 @@ def test_latest_is_downloaded(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
installer = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'-x64.bin')
installer = host.file('/media/atl/downloads/jira-core.'+upstream+'-x64.bin')
assert installer.exists
assert installer.user == 'root'
@@ -32,6 +32,6 @@ def test_completed_lockfile(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/jira-core.'+upstream+'-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

14
roles/product_install/molecule/jira_all/Dockerfile.j2 Normal file
View File

@@ -0,0 +1,14 @@
# Molecule managed
{% if item.registry is defined %}
FROM {{ item.registry.url }}/{{ item.image }}
{% else %}
FROM {{ item.image }}
{% endif %}
RUN if [ $(command -v apt-get) ]; then apt-get update && apt-get install -y python sudo bash ca-certificates && apt-get clean; \
elif [ $(command -v dnf) ]; then dnf makecache && dnf --assumeyes install python sudo python-devel python*-dnf bash && dnf clean all; \
elif [ $(command -v yum) ]; then yum makecache fast && yum install -y python sudo yum-plugin-ovl bash && sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf && yum clean all; \
elif [ $(command -v zypper) ]; then zypper refresh && zypper install -y python sudo bash python-xml && zypper clean -a; \
elif [ $(command -v apk) ]; then apk update && apk add --no-cache python sudo bash ca-certificates; \
elif [ $(command -v xbps-install) ]; then xbps-install -Syu && xbps-install -y python sudo bash ca-certificates && xbps-remove -O; fi

30
roles/product_install/molecule/jira_all/molecule.yml Normal file
View File

@@ -0,0 +1,30 @@
---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
platforms:
- name: amazon_linux2
image: amazonlinux:2
groups:
- aws_node_local
- name: ubuntu_lts
image: ubuntu:bionic
groups:
- aws_node_local
provisioner:
name: ansible
options:
skip-tags: runtime_pkg
lint:
name: ansible-lint
inventory:
links:
group_vars: ../../../../group_vars/
verifier:
name: testinfra
lint:
name: flake8
enabled: false

28
roles/product_install/molecule/jira_all/playbook.yml Normal file
View File

@@ -0,0 +1,28 @@
---
- name: Converge
hosts: all
vars:
atl_product_family: "jira"
atl_product_edition: "jira-software"
atl_product_user: "jira"
atl_product_version: "7.13.1"
atl_install_jsd_as_obr: true
atl_systemd_service_name: "jira.service"
atl_jdbc_encoding: 'UNICODE'
atl_jdbc_collation: 'C'
atl_jdbc_ctype: 'C'
atl_jdbc_template: 'template0'
pre_tasks:
- name: Create cache dir
file:
path: '/media/atl/jira/shared/'
state: directory
- name: Seed version
copy:
dest: '/media/atl/jira/shared/jira-core.version'
content: "7.13.1"
force: false # For idempotency check
roles:
- role: linux_common
- role: product_common
- role: product_install

48
roles/product_install/molecule/jira_all/tests/test_default.py Normal file
View File

@@ -0,0 +1,48 @@
import os
from six.moves import urllib
import testinfra.utils.ansible_runner
import json
testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')
def test_version_is_correct(host):
verfile = host.file('/media/atl/jira/shared/jira-software.version')
assert verfile.exists
assert verfile.content.decode("UTF-8").strip() == "7.13.1"
def test_is_downloaded(host):
installer = host.file('/media/atl/downloads/jira-software.7.13.1-x64.bin')
assert installer.exists
assert installer.user == 'root'
def test_completed_lockfile(host):
lockfile = host.file('/media/atl/downloads/jira-software.7.13.1-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'
def test_is_unpacked(host):
installer = host.file('/opt/atlassian/jira-software/7.13.1/atlassian-jira/')
assert installer.exists
assert installer.is_directory
assert installer.user == 'jira'
assert installer.mode == 0o0755
def test_obr_is_downloaded(host):
installer = host.file('/media/atl/downloads/jira-servicedesk-application-3.16.1.obr')
assert installer.exists
assert installer.user == 'root'
def test_obr_completed_lockfile(host):
lockfile = host.file('/media/atl/downloads/jira-servicedesk-application-3.16.1.obr_completed')
assert lockfile.exists
assert lockfile.user == 'root'
def test_obr_is_unpacked(host):
jsdjar = host.file('/media/atl/jira/shared/plugins/installed-plugins/jira-servicedesk-application-3.16.1.jar')
assert jsdjar.exists
assert jsdjar.user == 'jira'
assert jsdjar.mode == 0o0750

4
roles/product_install/molecule/jira_cached_with_downgrade/tests/test_default.py
View File

@@ -14,12 +14,12 @@ def test_version_is_correct(host):
assert verfile.content.decode("UTF-8").strip() == "7.10.2"
def test_is_downloaded(host):
installer = host.file('/media/atl/jira/shared/downloads/jira-core.7.10.2-x64.bin')
installer = host.file('/media/atl/downloads/jira-core.7.10.2-x64.bin')
assert installer.exists
assert installer.user == 'root'
def test_completed_lockfile(host):
lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.7.10.2-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/jira-core.7.10.2-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/jira_cached_with_upgrade/tests/test_default.py
View File

@@ -14,12 +14,12 @@ def test_version_is_correct(host):
assert verfile.content.decode("UTF-8").strip() == "7.10.1"
def test_is_downloaded(host):
installer = host.file('/media/atl/jira/shared/downloads/jira-core.7.10.1-x64.bin')
installer = host.file('/media/atl/downloads/jira-core.7.10.1-x64.bin')
assert installer.exists
assert installer.user == 'root'
def test_completed_lockfile(host):
lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.7.10.1-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/jira-core.7.10.1-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/jira_software_latest/tests/test_default.py
View File

@@ -35,7 +35,7 @@ def test_latest_is_downloaded(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
installer = host.file('/media/atl/jira/shared/downloads/jira-software.'+upstream+'-x64.bin')
installer = host.file('/media/atl/downloads/jira-software.'+upstream+'-x64.bin')
assert installer.exists
assert installer.user == 'root'
@@ -44,6 +44,6 @@ def test_completed_lockfile(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
lockfile = host.file('/media/atl/jira/shared/downloads/jira-software.'+upstream+'-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/jira-software.'+upstream+'-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/jira_tarball/tests/test_default.py
View File

@@ -23,7 +23,7 @@ def test_latest_is_downloaded(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
installer = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'.tar.gz')
installer = host.file('/media/atl/downloads/jira-core.'+upstream+'.tar.gz')
assert installer.exists
assert installer.user == 'root'
@@ -32,6 +32,6 @@ def test_completed_lockfile(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'.tar.gz_completed')
lockfile = host.file('/media/atl/downloads/jira-core.'+upstream+'.tar.gz_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/jira_version_from_file/tests/test_default.py
View File

@@ -14,12 +14,12 @@ def test_version_is_correct(host):
assert verfile.content.decode("UTF-8").strip() == "7.9.0"
def test_is_downloaded(host):
installer = host.file('/media/atl/jira/shared/downloads/jira-core.7.9.0-x64.bin')
installer = host.file('/media/atl/downloads/jira-core.7.9.0-x64.bin')
assert installer.exists
assert installer.user == 'root'
def test_completed_lockfile(host):
lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.7.9.0-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/jira-core.7.9.0-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/jira_version_latest/tests/test_default.py
View File

@@ -23,7 +23,7 @@ def test_latest_is_downloaded(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
installer = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'-x64.bin')
installer = host.file('/media/atl/downloads/jira-core.'+upstream+'-x64.bin')
assert installer.exists
assert installer.user == 'root'
@@ -32,6 +32,6 @@ def test_completed_lockfile(host):
upstream_json = json.load(upstream_fd)
upstream = upstream_json['version']
lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.'+upstream+'-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/jira-core.'+upstream+'-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/jira_version_override/tests/test_default.py
View File

@@ -14,12 +14,12 @@ def test_version_is_correct(host):
assert verfile.content.decode("UTF-8").strip() == "7.13.2"
def test_is_downloaded(host):
installer = host.file('/media/atl/jira/shared/downloads/jira-core.7.13.2-x64.bin')
installer = host.file('/media/atl/downloads/jira-core.7.13.2-x64.bin')
assert installer.exists
assert installer.user == 'root'
def test_completed_lockfile(host):
lockfile = host.file('/media/atl/jira/shared/downloads/jira-core.7.13.2-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/jira-core.7.13.2-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/servicedesk3/tests/test_default.py
View File

@@ -14,12 +14,12 @@ def test_version_is_correct(host):
assert verfile.content.decode("UTF-8").strip() == "3.9.0"
def test_is_downloaded(host):
installer = host.file('/media/atl/jira/shared/downloads/servicedesk.3.9.0-x64.bin')
installer = host.file('/media/atl/downloads/servicedesk.3.9.0-x64.bin')
assert installer.exists
assert installer.user == 'root'
def test_completed_lockfile(host):
lockfile = host.file('/media/atl/jira/shared/downloads/servicedesk.3.9.0-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/servicedesk.3.9.0-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/servicedesk4/tests/test_default.py
View File

@@ -14,12 +14,12 @@ def test_version_is_correct(host):
assert verfile.content.decode("UTF-8").strip() == "4.1.0"
def test_is_downloaded(host):
installer = host.file('/media/atl/jira/shared/downloads/servicedesk.4.1.0-x64.bin')
installer = host.file('/media/atl/downloads/servicedesk.4.1.0-x64.bin')
assert installer.exists
assert installer.user == 'root'
def test_completed_lockfile(host):
lockfile = host.file('/media/atl/jira/shared/downloads/servicedesk.4.1.0-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/servicedesk.4.1.0-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

4
roles/product_install/molecule/servicedesk_latest/tests/test_default.py
View File

@@ -23,12 +23,12 @@ def test_version_is_correct(host):
assert verfile.content.decode("UTF-8").strip() == sd
def test_is_downloaded(host):
installer = host.file('/media/atl/jira/shared/downloads/servicedesk.'+sd+'-x64.bin')
installer = host.file('/media/atl/downloads/servicedesk.'+sd+'-x64.bin')
assert installer.exists
assert installer.user == 'root'
def test_completed_lockfile(host):
lockfile = host.file('/media/atl/jira/shared/downloads/servicedesk.'+sd+'-x64.bin_completed')
lockfile = host.file('/media/atl/downloads/servicedesk.'+sd+'-x64.bin_completed')
assert lockfile.exists
assert lockfile.user == 'root'

1
roles/product_install/tasks/jira-all_extra_tasks.yml Symbolic link
View File

@@ -0,0 +1 @@
no_op.yml

174
roles/product_install/tasks/jira-servicedesk_as_obr.yml Normal file
View File

@@ -0,0 +1,174 @@
---
- name: Get the installer product version info
uri:
url: "{{ atl_mpac_products }}/key/jira-software/versions/name/{{ atl_product_version }}"
return_content: yes
register: atl_product_version_info
- name: Show the returned build number
debug:
msg="buildNumber={{ atl_product_version_info.json.buildNumber }}"
- name: Get the JSD build version info
uri:
url: "{{ atl_mpac_products }}/key/jira-servicedesk/versions/latest?application=\
jira&applicationBuild={{ atl_product_version_info.json.buildNumber }}"
return_content: yes
register: atl_jsd_build_info
- name: Show the returned obr binary href
debug:
msg="obr_ref={{ atl_jsd_build_info.json._embedded.artifact._links.binary.href }}"
- name: how about getting the obr filename
debug:
msg="obr_name=jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr"
- name: is shared_home set ?
debug:
msg="atl_product_home_shared_download_dir={{ atl_product_home_shared_download_dir }}"
# For the first run a temp obr should be downloaded but moved to
# shared home to ensure all subsequent nodes have access
# to the same specific version binary.
# To prevent a race condition with multiple downloads at the same time
# a directory is used as a lockfile (atomic operation) when moving obr.
- name: Set assumptions to avoid race condition
set_fact:
download_obr: true
move_obr: false
atl_obr_download_href: "{{ atl_jsd_build_info.json._embedded.artifact._links.binary.href }}"
atl_obr_filename: "jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr"
atl_obr_download: "{{ atl_installer_temp }}/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr"
atl_obr_shared_download: "{{ atl_product_home_shared_download_dir }}/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr"
atl_obr_moving_lock: "{{ atl_product_home_shared_download_dir }}/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr_moving"
atl_obr_completed_lock: "{{ atl_product_home_shared_download_dir }}/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.obr_completed"
# Check for pre-downloaded obr on shared_home and completed lock dir.
- name: Check for completed lock directory
stat:
path: "{{ atl_obr_completed_lock }}"
register: completed_lock
- name: Check for obr in home_shared
stat:
path: "{{ atl_obr_shared_download }}"
register: home_shared_download
# If obr exists and lockdir exists use this obr instead
- name: Check lock directory and obr exists on shared_home
set_fact:
download_obr: false
atl_obr_download: "{{ atl_obr_shared_download }}"
when:
- home_shared_download.stat.exists
- completed_lock.stat.isdir is defined
- completed_lock.stat.isdir
# Fetch obr if required
- name: download_obr is true so fetch and do all the things
block:
# Fetch obr and copy to temp
- name: Fetch obr
get_url:
url: "{{ atl_obr_download_href }}"
dest: "{{ atl_obr_download }}"
mode: 0755
force: false
register: atl_obr_completed
# If obr was fetched make the lock directory
- name: Create moving_lock.
file:
path: "{{ atl_obr_moving_lock }}"
state: directory
when:
- atl_obr_completed is succeeded
register: moving_lock_created
# Directory lock was created by this run?
# If so, then set a fact intending to move obr
- name: Move obr Scenario - lock created by this run
set_fact:
move_obr: true
when:
- moving_lock_created is succeeded
- moving_lock_created.changed
# Otherwise directory lock was either already created or
# could not be created. Fall back is to continue and install from temp
when: download_obr
# If the intention is to move obr to home_shared
- name: Move obr to home_shared
block:
- name: Copy temp installer to home_shared
copy:
src: "{{ atl_obr_download }}"
dest: "{{ atl_obr_shared_download }}"
remote_src: true
register: copied
- name: Create completed_lock once obr downloaded and copied
file:
path: "{{ atl_obr_completed_lock }}"
state: directory
when: copied is succeeded
register: completed_lock_created
- name: Remove moving_lock to show that obr is completed
file:
path: "{{ atl_obr_moving_lock }}"
state: absent
when:
- completed_lock_created is succeeded
- copied is succeeded
register: moving_lock_removed
- name: Delete old temp installer
file:
path: "{{ atl_obr_download }}"
state: absent
when: moving_lock_removed is succeeded
register: temp_deleted
- name: Set install to home_shared location
set_fact:
atl_obr_download: "{{ atl_obr_shared_download }}"
when: temp_deleted is succeeded
when: move_obr
# At this point the binary is in {{ atl_obr_download }}
# (which is either on home_shared or temp)
- name: Ensure instaled-plugins dir exists
file:
path: "{{ atl_product_home_shared }}/plugins/installed-plugins"
state: directory
mode: 0750
owner: "{{ atl_product_user }}"
group: "{{ atl_product_user }}"
# Note as ansible unarchive cant handle "-j junk paths" we need to ignore errors to bypass the path verify
- name: Unpack the obr into the installed-plugins dir
unarchive:
remote_src: true
src: "{{ atl_obr_download }}"
dest: "{{ atl_product_home_shared }}/plugins/installed-plugins"
creates: "{{ atl_product_home_shared }}/plugins/installed-plugins/jira-servicedesk-application-{{ atl_jsd_build_info.json.name }}.jar"
list_files: no
exclude:
- M*
owner: "{{ atl_product_user }}"
group: "{{ atl_product_user }}"
mode: 0750
ignore_errors: yes
- name: Move JSD dependency jars into the installed-plugins dir
shell:
cmd: "mv {{ atl_product_home_shared }}/plugins/installed-plugins/dependencies/*.jar {{ atl_product_home_shared }}/plugins/installed-plugins"
creates: "{{ atl_product_home_shared }}/plugins/installed-plugins/servicedesk-core-ui-plugin-{{ atl_jsd_build_info.json.name }}-*.jar"

6
roles/product_install/tasks/main.yml
View File

@@ -249,3 +249,9 @@
dest: "{{ atl_product_installation_current }}"
state: link
force: true
- name: Include if jsd is requested to be installed from OBR
include_tasks: "jira-servicedesk_as_obr.yml"
when:
- atl_install_jsd_as_obr is defined
- atl_install_jsd_as_obr

14
roles/restore_backups/molecule/restore/Dockerfile.j2 Normal file
View File

@@ -0,0 +1,14 @@
# Molecule managed
{% if item.registry is defined %}
FROM {{ item.registry.url }}/{{ item.image }}
{% else %}
FROM {{ item.image }}
{% endif %}
RUN if [ $(command -v apt-get) ]; then apt-get update && apt-get install -y python sudo bash ca-certificates && apt-get clean; \
elif [ $(command -v dnf) ]; then dnf makecache && dnf --assumeyes install python sudo python-devel python*-dnf bash && dnf clean all; \
elif [ $(command -v yum) ]; then yum makecache fast && yum install -y python sudo yum-plugin-ovl bash && sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf && yum clean all; \
elif [ $(command -v zypper) ]; then zypper refresh && zypper install -y python sudo bash python-xml && zypper clean -a; \
elif [ $(command -v apk) ]; then apk update && apk add --no-cache python sudo bash ca-certificates; \
elif [ $(command -v xbps-install) ]; then xbps-install -Syu && xbps-install -y python sudo bash ca-certificates && xbps-remove -O; fi

30
roles/restore_backups/molecule/restore/molecule.yml Normal file
View File

@@ -0,0 +1,30 @@
---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
platforms:
- name: amazon_linux2
image: amazonlinux:2
groups:
- aws_node_local
ulimits:
- nofile:262144:262144
provisioner:
name: ansible
options:
skip-tags: runtime_pkg
lint:
name: ansible-lint
options:
x: ["701"]
inventory:
links:
group_vars: ../../../../group_vars/
verifier:
name: testinfra
lint:
name: flake8
enabled: false

75
roles/restore_backups/molecule/restore/playbook.yml Normal file
View File

@@ -0,0 +1,75 @@
---
- name: Converge
hosts: all
vars:
atl_backup_home_dest: "{{ test_archive }}"
atl_backup_home_restore_canary_path: '/tmp/canary.tmp'
atl_backup_id: 'test-backup'
atl_backup_manifest_url: 'fake_manifest'
atl_product_edition: 'jira-software'
atl_product_home_shared: '/media/atl/jira/shared'
atl_product_user: 'jira'
atl_product_user_uid: '2001'
atl_product_version_cache: "{{ atl_product_home_shared }}/{{ atl_product_edition }}.version"
test_archive: '/tmp/hello.tar.gz'
test_archive_file: 'hello.txt'
test_archive_source: '/tmp/hello'
test_pre_step_prefix: '[PRE-TEST]'
test_product_version_file: "/tmp/{{ atl_product_edition }}.version"
pre_tasks:
- name: "{{ test_pre_step_prefix }} Install tar"
package:
state: present
name: tar
- name: "{{ test_pre_step_prefix }} Install useradd and groupadd binaries"
package:
state: present
name: shadow-utils
- name: "{{ test_pre_step_prefix }} Create application group"
group:
name: "{{ atl_product_user }}"
gid: "{{ atl_product_user_uid }}"
- name: "{{ test_pre_step_prefix }} Create application user"
user:
name: "{{ atl_product_user }}"
uid: "{{ atl_product_user_uid }}"
group: "{{ atl_product_user }}"
- block:
- name: "{{ test_pre_step_prefix }} Create a directory for the shared home archive"
file:
path: "{{ test_archive_source }}"
state: directory
mode: 0755
- name: "{{ test_pre_step_prefix }} Create a file in the shared home"
lineinfile:
create: yes
line: 'Hello, world!'
path: "{{ test_archive_source }}/{{ test_archive_file }}"
mode: 0640
- name: "{{ test_pre_step_prefix }} Create the version file in the shared home"
lineinfile:
create: yes
line: '8.5'
path: "{{ test_product_version_file }}"
mode: 0640
- name: "{{ test_pre_step_prefix }} Archive the shared home"
archive:
path:
- "{{ test_archive_source }}"
- "{{ test_product_version_file }}"
dest: "{{ test_archive }}"
owner: "{{ atl_product_user }}"
tasks:
- name: Install distro-specific restore support packages
include_tasks: "../../tasks/{{ ansible_distribution|lower }}.yml"
- name: Restore shared home
include_tasks: "../../tasks/home_restore.yml"

39
roles/restore_backups/molecule/restore/tests/test_default.py Normal file
View File

@@ -0,0 +1,39 @@
import os
import pytest
import testinfra.utils.ansible_runner
testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')
@pytest.mark.parametrize('exe', [
'/usr/bin/pg_dump',
'/usr/bin/pg_restore',
'/usr/bin/psql'
])
def test_postgresql_amazon_linux_extras_exes(host, exe):
assert host.file(exe).exists
def test_postgresql_version(host):
pg_dump_version_output = host.check_output('pg_dump --version')
assert '(PostgreSQL) 9.6' in pg_dump_version_output
@pytest.mark.parametrize('file', [
'/media/atl/jira/shared',
'/media/atl/jira/shared/hello',
'/media/atl/jira/shared/hello/hello.txt'
])
def test_shared_home_owner(host, file):
assert host.file(file).exists
assert host.file(file).user == 'jira'
assert host.file(file).group == 'jira'
def test_file_modes(host):
assert host.file('/media/atl/jira/shared/hello').mode == 0o755
assert host.file('/media/atl/jira/shared/hello/hello.txt').mode == 0o640
def test_version_file_owned_by_root(host):
assert host.file('/media/atl/jira/shared/jira-software.version').exists
assert host.file('/media/atl/jira/shared/jira-software.version').user == 'root'
assert host.file('/media/atl/jira/shared/jira-software.version').group == 'root'

2
roles/restore_backups/tasks/amazon.yml
View File

@@ -8,3 +8,5 @@
PYTHON: /bin/python
args:
creates: /usr/bin/psql
environment:
PYTHON: /bin/python

44
roles/restore_backups/tasks/home_restore.yml Normal file
View File

@@ -0,0 +1,44 @@
---
- name: Check for the restore canary file
stat:
path: "{{ atl_backup_home_restore_canary_path }}"
register: restore_canary
- block:
- name: Create shared home if necessary
file:
path: "{{ atl_product_home_shared }}"
state: directory
mode: 0750
owner: "{{ atl_product_user }}"
group: "{{ atl_product_user }}"
- name: Restore the shared-home backup
unarchive:
src: "{{ atl_backup_home_dest }}"
remote_src: yes
dest: "{{ atl_product_home_shared }}"
- name: Set shared home owner and group to application user
file:
path: "{{ atl_product_home_shared }}"
recurse: yes
group: "{{ atl_product_user }}"
owner: "{{ atl_product_user }}"
state: directory
- name: Set version file owner and group to root
file:
path: "{{ atl_product_version_cache }}"
group: root
owner: root
state: file
# Ignore the error in case there is no product version file in the backup
ignore_errors: yes
- name: Create restore-canary if necessary
copy:
dest: "{{ atl_backup_home_restore_canary_path }}"
content: "{{ atl_backup_id }}"
when: not restore_canary.stat.exists

40
roles/restore_backups/tasks/main.yml
View File

@@ -68,7 +68,9 @@
overwrite: different
bucket: "{{ atl_backup_manifest.artifacts.db.location.location | urlsplit('hostname') }}"
object: "{{ atl_backup_manifest.artifacts.db.location.location | urlsplit('path') }}"
dest: "{{ atl_backup_db_dest }}"
# We save the backup as a .tar file so that the postgresql_db module uses pg_restore instead of psql to do restore
# This can be removed when ansible 2.10 is released
dest: "{{ atl_backup_db_dest }}.tar"
- name: Fetch Home backup from S3
aws_s3:
@@ -96,42 +98,18 @@
template: "{{ atl_jdbc_template }}"
# Depends on fetch_backup roles
state: restore
target: "{{ atl_backup_db_dest }}"
target: "{{ atl_backup_db_dest }}.tar"
target_opts: "-Fc"
register: result
# managed DBs in cloud providers are not allowing full root access to the DB engine, we can safely ignore the COMMENT ON EXTENSION error
failed_when:
- result.rc != 0
- "COMMENT ON EXTENSION" not in result.msg
- '"COMMENT ON EXTENSION" not in result.msg'
when: db_created.changed and atl_backup_db_dest is defined
- name: Check for the restore canary file
stat:
path: "{{ atl_backup_home_restore_canary_path }}"
register: restore_canary
- block:
- name: Create shared home if necessary
file:
path: "{{ atl_product_home_shared }}"
state: directory
mode: 0750
owner: "{{ atl_product_user }}"
group: "{{ atl_product_user }}"
- name: Restore the shared-home backup
unarchive:
src: "{{ atl_backup_home_dest }}"
dest: "{{ atl_product_home_shared }}"
owner: "{{ atl_product_user }}"
group: "{{ atl_product_user }}"
- name: Create restore-canary if necessary
copy:
dest: "{{ atl_backup_home_restore_canary_path }}"
content: "{{ atl_backup_id }}"
when: not restore_canary.stat.exists
- name: Restore shared home
include_tasks: "home_restore.yml"
when: atl_restore_required