Merged in DCD-621-parallelize-builds (pull request #20)

DCD-621 parallelize builds

* DCD-621: Parallelizes tests ny invoking the parralelize script

* DCD-621: Removes leading / to start jira command in product_startup role

* DCD-621: Runs a scenario by name instead of using the all parameter

* DCD-621: A better find all scenarios function to ensure we look at relevant molecule scenarios. Better logging too

* DCD-510: Parallelizes tests further, running in 26 batches on 1 each. The pipeline file is now generated using Jinja

* DCD-510: Adds a pre check stage to verify if number of batches match actual number of scenarios. This test will fail (deliberately)

* DCD-510: Adds a pre check stage to verify if number of batches match actual number of scenarios. Fixes 'test' failure

* DCD-590: Adds note to development.md on how to generate a pipeline file. Adds a makefile and updates a few script commands

* DCD-590: Better documentation in development README document. Updates YML with instructions on how to fix issues with the prevalidation stage

Approved-by: Steve Smith <ssmith@atlassian.com>
Approved-by: Ben Partridge <bpartridge@atlassian.com>

DEVELOPMENT.md

@@ -46,7 +46,7 @@ development environment:
 
 ### Step 1.4: Run some tests against a role
 
-Molecule is a testing framework for Ansible. We use this to test the
+[Molecule](https://molecule.readthedocs.io/en/stable/) is a testing framework for Ansible. We use this to test the
 functionality of individual and groups of roles, and to ensure cross-platform
 compatibility (currently Amazon Linux 2 and Ubuntu LTS).
 
@@ -133,3 +133,16 @@ Some notable files and locations:
   stdout can be viewed with systemctl status jira.service and journalctl -u
   jira.service.
 * Other Tomcat logs can be located under /opt/atlassian/jira/current/logs/...
+
+## Running molecule tests on CI
+* This repository is configured to run tests on [bitbucket pipelines](https://bitbucket.org/atlassian/dc-deployments-automation/addon/pipelines/home).
+    * Pipeline definition is located in the [root of the repository](https://bitbucket.org/atlassian/dc-deployments-automation/src/master/bitbucket-pipelines.yml)
+    * Pipeline configuration is generated using Jinja2 and a simple python script. Pipeline generator is located [here](https://bitbucket.org/atlassian/dc-deployments-automation/src/master/pipeline_generator/)
+    * A pipeline configuration is generated by running the following make command 
+    ```
+     make generate-pipeline > ../bitbucket-pipelines.yml
+    ```  
+    
+* [Molecule](https://molecule.readthedocs.io/en/stable/) tests are run in batches. A single test is run per batch to optimize for a faster dev feedback loop
+
+* If you create a new role or add a new molecule scenario, then please ensure that you generate a new pipeline configuration by running the make command described above. If a new pipeline configuration is not generated, then the CI may not run any tests as it would fail at a pre-test validate stage.

Pipfile

@@ -11,6 +11,7 @@ molecule = "==2.20.1"
 docker = "==4.0.1"
 six = "*"
 taskcat = "*"
+Jinja2 = "*"
 
 [requires]
 python_version = "2.7"

bin/run-tests-in-batches

@@ -6,7 +6,7 @@ set -e
 # Install dev packages from Pipfile.lock if necessary.
 pipenv sync --dev
 
-BATCH_SIZE=3
+BATCH_SIZE=1
 BATCH_NUMBER=""
 batch_args="$1"
 
@@ -29,17 +29,25 @@ case ${BATCH_NUMBER} in
     *) echo "Bad Input for Batch number. ${BATCH_NUMBER} is not a valid batch number (Should be a number >= 1)" && exit -1 ;;
 esac
 
-scenarios=( $(ls -d1 roles/**/molecule/* | sort) )
+scenarios=( $(find roles -type f -name 'molecule.yml' -exec dirname {} ';' | sort) )
 
 offset=$(( ${BATCH_NUMBER} - 1))
 test_start_index=$(( ${offset} * $BATCH_SIZE ))
 scenario_batch="${scenarios[@]:$test_start_index:$BATCH_SIZE}"
 
-echo "Scenarios that will be executed as part of this batch: ${scenario_batch}"
+if [[ ${scenario_batch[@]} ]]; then
+    echo "Scenarios that will be executed as part of this batch: ${scenario_batch}"
+else
+    echo "Scenario Batch for ${BATCH_NUMBER} is empty. Exiting (Non zero) now"
+    exit -1
+fi
 
 for scenario in ${scenario_batch}; do
-    pushd $(dirname $(dirname $scenario))
+    scenario_name=$(basename ${scenario})
+    role_name=$(dirname $(dirname ${scenario}))
+    echo "Running scenario ${scenario_name} in ${role_name}"
+    pushd ${role_name}
     pipenv run \
-       molecule test --all
+       molecule test -s ${scenario_name}
     popd
 done;

bitbucket-pipelines.yml

@@ -14,78 +14,251 @@ pipelines:
     - step:
         name: Pre Parallelization stage
         script:
-          - echo "Running tests in 9 batches of 3 scenarios (max) each"
+          - echo "Running tests in 26 batches"
+    - step:
+        name: Check if number of batches match actual number of scenarios
+        script:
+          - |
+              actual_scenario_count=$(find ./roles -type f -name "molecule.yml" -exec dirname {} ';' | wc -l |  sed -e 's/^[[:space:]]*//')
+              grep  "Running tests in ${actual_scenario_count} batches" bitbucket-pipelines.yml
+              GREP_RETURN_CODE=$?
+              if [[ $GREP_RETURN_CODE -ne 0 ]]; then
+                  echo "Mismatch between expected and actual number [${actual_scenario_count}] of scenarios. Please look at https://bitbucket.org/atlassian/dc-deployments-automation/src/master/DEVELOPMENT.md for instructions on how to fix this error."
+                  exit $GREP_RETURN_CODE
+              fi
     - parallel:
        - step:
-            name: Molecule test batch 1
+           name: Molecule Test Batch 1
            services:
              - docker
            script:
            - apt-get update && apt-get install -y virtualenv python-dev
            - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 1
-        - step:
-            name: Molecule test batch 2
-            services:
-              - docker
-            script:
-              - apt-get update && apt-get install -y virtualenv python-dev
-              - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 2
-        - step:
-            name: Molecule test batch 3
-            services:
-              - docker
-            script:
-              - apt-get update && apt-get install -y virtualenv python-dev
-              - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 3
-        - step:
-            name: Molecule test batch 4
-            services:
-              - docker
-            script:
-              - apt-get update && apt-get install -y virtualenv python-dev
-              - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 4
-        - step:
-            name: Molecule test batch 5
-            services:
-              - docker
-            script:
-              - apt-get update && apt-get install -y virtualenv python-dev
-              - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 5
-        - step:
-            name: Molecule test batch 6
-            services:
-              - docker
-            script:
-              - apt-get update && apt-get install -y virtualenv python-dev
-              - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 6
-        - step:
-            name: Molecule test batch 7
-            services:
-              - docker
-            script:
-              - apt-get update && apt-get install -y virtualenv python-dev
-              - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 7
-        - step:
-            name: Molecule test batch 8
-            services:
-              - docker
-            script:
-              - apt-get update && apt-get install -y virtualenv python-dev
-              - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 8
-        - step:
-            name: Molecule test batch 9
-            services:
-              - docker
-            script:
-              - apt-get update && apt-get install -y virtualenv python-dev
-              - ./bin/install-ansible
-              - ./bin/run-all-tests --batch 9
+           - ./bin/run-tests-in-batches --batch 1
+           
+       - step:
+           name: Molecule Test Batch 2
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 2
+           
+       - step:
+           name: Molecule Test Batch 3
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 3
+           
+       - step:
+           name: Molecule Test Batch 4
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 4
+           
+       - step:
+           name: Molecule Test Batch 5
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 5
+           
+       - step:
+           name: Molecule Test Batch 6
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 6
+           
+       - step:
+           name: Molecule Test Batch 7
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 7
+           
+       - step:
+           name: Molecule Test Batch 8
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 8
+           
+       - step:
+           name: Molecule Test Batch 9
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 9
+           
+       - step:
+           name: Molecule Test Batch 10
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 10
+           
+       - step:
+           name: Molecule Test Batch 11
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 11
+           
+       - step:
+           name: Molecule Test Batch 12
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 12
+           
+       - step:
+           name: Molecule Test Batch 13
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 13
+           
+       - step:
+           name: Molecule Test Batch 14
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 14
+           
+       - step:
+           name: Molecule Test Batch 15
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 15
+           
+       - step:
+           name: Molecule Test Batch 16
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 16
+           
+       - step:
+           name: Molecule Test Batch 17
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 17
+           
+       - step:
+           name: Molecule Test Batch 18
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 18
+           
+       - step:
+           name: Molecule Test Batch 19
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 19
+           
+       - step:
+           name: Molecule Test Batch 20
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 20
+           
+       - step:
+           name: Molecule Test Batch 21
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 21
+           
+       - step:
+           name: Molecule Test Batch 22
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 22
+           
+       - step:
+           name: Molecule Test Batch 23
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 23
+           
+       - step:
+           name: Molecule Test Batch 24
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 24
+           
+       - step:
+           name: Molecule Test Batch 25
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 25
+           
+       - step:
+           name: Molecule Test Batch 26
+           services:
+             - docker
+           script:
+           - apt-get update && apt-get install -y virtualenv python-dev
+           - ./bin/install-ansible
+           - ./bin/run-tests-in-batches --batch 26
+           
        

pipeline_generator/Makefile

@@ -0,0 +1,2 @@
+generate-pipeline:
+	@python pipeline.py

pipeline_generator/pipeline.py

@@ -0,0 +1,62 @@
+from jinja2 import Template
+from pathlib import Path
+import os
+
+PIPELINE_TEMPLATE_J2_FILE = 'templates/bitbucket-pipelines.yml.j2'
+ROLES_DIR = 'roles/'
+
+
+class Pipeline:
+    def generate_pipeline(self):
+        template_string = self._load_template_as_string()
+        template = Template(template_string)
+        steps = self._build_steps()
+        generated_output = template.render(parallel_steps=steps)
+        print(generated_output)
+
+    def _build_steps(self):
+        return [Step(f"Molecule Test Batch {index}", self._build_script_commands(index)) for index, scenario in
+                enumerate(self._find_all_scenarios(), 1)]
+
+    @staticmethod
+    def _build_script_commands(index):
+        return ScriptCommand(f"./bin/run-tests-in-batches --batch {index}").all_commands()
+
+    @staticmethod
+    def _find_all_scenarios():
+        scenario_dirs = []
+        for root, dirs, files in os.walk(Path(os.path.join(os.path.dirname(__file__), "..", ROLES_DIR))):
+            [scenario_dirs.append(root) for f in files if f.endswith("molecule.yml")]
+        return scenario_dirs
+
+    @staticmethod
+    def _load_template_as_string():
+        path = Path(os.path.join(os.path.dirname(__file__), PIPELINE_TEMPLATE_J2_FILE))
+        return path.read_text()
+
+
+class Step:
+    def __init__(self, name, script_commands=None):
+        if script_commands is None:
+            script_commands = []
+        self.name = name
+        self.scriptCommands = script_commands
+
+
+class ScriptCommand:
+    INSTALL_PACKAGES_COMMAND = "apt-get update && apt-get install -y virtualenv python-dev"
+    INSTALL_ANSIBLE_COMMAND = "./bin/install-ansible"
+
+    def __init__(self, test_command):
+        self.test_command = test_command
+
+    def all_commands(self):
+        return [self.INSTALL_PACKAGES_COMMAND, self.INSTALL_ANSIBLE_COMMAND, self.test_command]
+
+
+def main():
+    Pipeline().generate_pipeline()
+
+
+if __name__ == '__main__':
+    main()

pipeline_generator/templates/bitbucket-pipelines.yml.j2

@@ -0,0 +1,39 @@
+---
+
+image: atlassian/default-image:2
+options:
+  size: 2x
+
+definitions:
+  services:
+    docker:
+      memory: 4096
+
+pipelines:
+  default:
+    - step:
+        name: Pre Parallelization stage
+        script:
+          - echo "Running tests in {{ parallel_steps|length }} batches"
+    - step:
+        name: Check if number of batches match actual number of scenarios
+        script:
+          - |
+              actual_scenario_count=$(find ./roles -type f -name "molecule.yml" -exec dirname {} ';' | wc -l |  sed -e 's/^[[:space:]]*//')
+              grep  "Running tests in ${actual_scenario_count} batches" bitbucket-pipelines.yml
+              GREP_RETURN_CODE=$?
+              if [[ $GREP_RETURN_CODE -ne 0 ]]; then
+                  echo "Mismatch between expected and actual number [${actual_scenario_count}] of scenarios. Please look at https://bitbucket.org/atlassian/dc-deployments-automation/src/master/DEVELOPMENT.md for instructions on how to fix this error."
+                  exit $GREP_RETURN_CODE
+              fi
+    - parallel:
+       {% for parallel_step in parallel_steps -%}
+        - step:
+           name: {{ parallel_step.name }}
+           services:
+             - docker
+           script:
+           {% for scriptCommand in parallel_step.scriptCommands -%}
+              - {{ scriptCommand }}
+           {% endfor %}
+       {% endfor %}

roles/product_startup/defaults/main.yml

@@ -5,7 +5,7 @@ atl_startup_enable: true
 atl_startup_restart: "{{ lookup('env', 'ATL_STARTUP_RESTART') or true }}"
 
 atl_startup_script_map:
-  jira: "/bin/start-jira.sh"
+  jira: "bin/start-jira.sh"
   confluence: "bin/start-confluence.sh"
   stash: "bin/start-bitbucket.sh"
   crowd: "start_crowd.sh"