Full Integration of Jenkins and Gerrit 65/6265/2
authorTrevor Bramwell <tbramwell@linuxfoundation.org>
Sun, 13 Aug 2017 06:27:46 +0000 (23:27 -0700)
committerTrevor Bramwell <tbramwell@linuxfoundation.org>
Thu, 7 Sep 2017 19:47:36 +0000 (12:47 -0700)
- Run exclusive services with NGINX

  In favor of writing individual configs for NGINX, and since all we're
  doing is reverse proxying, using jwilder/nginx-proxy and setting
  VIRTUAL_HOST should be all we need to get NGINX working and allows for
  individual services to be brought up without causing NGINX to fall
  over due the other upstream not being online.

- Config Jenkins & Gerrit integration through several Groovy scripts

  These scripts provide all the initial configuration needed to get
  Jenkins and Gerrit talking to each other. They include:

  - Disabling the CLI over remoting
  - Enabling CSRF protection
  - Enabling the Agent -> Master subsystem
  - Setting the Jenkins hostname
  - Creating SSH credentials for connecting Jenkins to Gerrit
  - Configuring the Gerrit-Trigger plugins
  - Adding a single Jenkins SSH agent
  - Creating a JJB ini file through the Config Files plugin that Jenkins
    agent will use to configure JJB
  - Setting the global environment variable GIT_URL requires by
    Global-JJB

- Initialization Container

  After the Jenkins and Gerrit are some more work is needed to allow
  them to interact such as:

  - Creating an ssh key in Jenkins home directory
  - Creating a user in Gerrit that Jenkins will use to access the
    event-stream over ssh
  - Writing out the basic configuration for a ci-management repository
  - Bootstrapping JJB jobs into Jenkins
  - Adding the Verified label to Gerrit

  These are all done outside of the context of the Gerrit and Jenkins
  containers because most of them rely on processes already running, and
  allow us to use unmodified upstream containers.

- Updating plugins required by Global-JJB

Change-Id: I13cda69989c4832d451be1b841d177c9fdaf9e27
Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
20 files changed:
README.rst
config/gerrit.env
config/jenkins.env
config/ldap/bootstrap/groups.ldif
config/nginx/gerrit.conf [deleted file]
config/nginx/jenkins.conf [deleted file]
docker-compose.yml
init/Dockerfile [new file with mode: 0644]
init/config-ci-environment.sh [new file with mode: 0644]
jenkins/Dockerfile
jenkins/agent/Dockerfile [new file with mode: 0644]
jenkins/authorization.groovy [deleted file]
jenkins/basic-security-setup.groovy [new file with mode: 0644]
jenkins/create-jenkins-agent.groovy [new file with mode: 0644]
jenkins/create-jjbini.groovy [new file with mode: 0644]
jenkins/gerrit-config.groovy [new file with mode: 0644]
jenkins/global-properties.groovy [new file with mode: 0644]
jenkins/hostname.groovy [deleted file]
jenkins/ldap.groovy
jenkins/plugins.txt

index 3c1dac5..a7a0a17 100644 (file)
@@ -26,28 +26,34 @@ containers.
 Quick Reference
 ---------------
 
-Jenkins
-  localhost:8001
+Add the following to /etc/hosts::
 
-Gerrit
-  localhost:8000
-  localhost:29418
+  127.0.1.1 jenkins.localhost
+  127.0.1.2 gerrit.localhost
 
 Default user account: sandbox/sandbox
 
 Getting Started
 ---------------
 
+Add the following to /etc/hosts::
+
+  127.0.1.1 jenkins.localhost
+  127.0.1.2 gerrit.localhost
+
+.. Note: This is the same as setting the 'Host' header when sending a GET
+   request to localhost: `curl -H "Host: gerrit.localhost" localhost`
+
 .. code-block::
 
   docker-compose up -d
 
 Will bring up an environment containing all the services with
 authentication backed by LDAP, a simple ci-management repo in
-Gerrit[TODO], and a basic job in Jenkins that verifies commits to the
-ci-management repo[TODO].
+Gerrit, and a basic job in Jenkins that verifies commits to the
+ci-management repo.
 
-To bring up a single service in the foreground you can use[TODO - Nginx]:
+To bring up a single service in the foreground you can use:
 
 .. code-block::
 
@@ -75,20 +81,53 @@ For other useful docker-compose commands such as logs, see::
 
   docker-compose -h
 
-Default User
-------------
+Next Steps
+----------
 
-The default username and password in LDAP are:
+Once the environment is up and running, copy your ssh public-key and add
+it to the sandbox user in Gerrit. This can be either be done through the
+web interface or from the commandline::
 
- user: sandbox
- pass: sandbox
+  curl -L -X POST -u "sandbox:sandbox" -H "Content-type:text/plain" \
+    -d @$HOME/.ssh/id_rsa.pub http://gerrit.localhost/a/accounts/self/sshkeys/
 
-This user is part of the 'sandbox-admins' group and should have full
-admin rights on all services.
+.. note: It's important here the Content-type header is set, as Gerrit
+   always expects JSON, and URLs must end in '/'
 
-Goals
+Then you can clone the ci-management repo and modify it to your hearts
+content::
+
+  git clone ssh://sandbox@gerrit.localhost:29418/ci-management.git
+
+Set the gitreview username::
+
+  git config --add gitreview.username "sandbox"
+
+And ensure the Change-Id hook exists::
+
+  git review -s
+
+Putting up a patchset for review that modifies "\*.yaml" files should
+trigger the ci-management-jjb-verify job and add a -1/+1 Verified vote.
+
+Notes
 -----
 
+Init Container
+~~~~~~~~~~~~~~
+
+In order to fully configure both Jenkins and Gerrit, another container
+'init' is added as part of the startup to generate ssh keys, create the
+ci-management repo, configure users, and push the ci-management jobs to
+Jenkins.
+
+This is done in a weakly idempotent fashion by creating files after the
+command execute successfuly, so that if the environment is restarted the
+container doesn't die or modify existing data.
+
+Goals
+~~~~~
+
 The goal of this project is to have an easily created workshop where
 releng work can be tested or proof-of-concepts created.
 
@@ -101,17 +140,19 @@ Some examples:
  * Gerrit/LDAP group integration
 
 TODO
-----
+~~~~
 
 The following is a list of automation tasks still needed before the
-environment can be considered usable:
+environment can be considered stable:
 
-Nginx:
-- [ ] Configure NGINX container to use environment variables_
-      Note: If both Jenkins and Gerrit aren't both started, the NGINX
-      container will continuously restart. This can be worked around by
-      disabling the other service you don't want to use:
-      `mv nginx/config/gerrit.conf gerrit.conf.disabled`
+General:
+- [ ] Replace 'sandbox' names with 'workshop' since sandbox was just a
+      placeholder
+- [ ] Setup OpenLDAP over SSL by default
+- [ ] Collapse environment config into single file and add lots of
+      comments, so users don't need to track down the correct file
+- [ ] Make things more configurable. There are a lot of hardcoded names
+      in Groovy scripts which could be pulled from environment variables
 
 Nexus:
 - [ ] Setup and configure Nexus
@@ -120,22 +161,17 @@ Gerrit:
 - [ ] Remove postgres container configuration and replace with MariaDB
   (or make optional)
 
-General:
-- [ ] Setup OpenLDAP over SSL by default
-- [ ] Create a basic ci-management repository in Gerrit
-- [ ] Connect and configure Gerrit and Jenkins automatically
-  - [ ]  Have the Gerrit configuration setup in Jenkins
-  - [ ]  Create Gerrit user with ssh pubkey from Jenkins
-
 Jenkins:
 - [ ] Fix (on Jenkins restart)::
       WARNING: Caught exception evaluating:
       instance.hasExplicitPermission(attrs.sid,p) in /configureSecurity/.
       Reason: java.lang.NullPointerException
-- [ ] Create a basic SSH Jenkins agent that jobs can be ran on
-- [ ] Disable CLI remoting
-- [ ] Enable Agent -> Master Access Controls
+- [ ] Make Groovy scripts Idempotent
+
+Init:
+- [ ] Make steps strongly idempotent (verify the state they modify)
 
 .. _environment: https://docs.docker.com/compose/environment-variables/#configuring-compose-using-environment-variables
 .. _variables: https://docs.docker.com/samples/nginx/#using-environment-variables-in-nginx-configuration
 .. _openfrontier: https://github.com/openfrontier/ci-compose
+.. _jwilder/nginx-proxy: https://github.com/jwilder/nginx-proxy
index e25e5a4..2a90285 100644 (file)
 ##############################################################################
 
 #GERRIT_INIT_ARGS=" \
+#  --install-plugin=download-commands \
 #  --install-plugin=commit-message-length-validator \
 #  --install-plugin=replication \
-#"
 #  --install-plugin=delete-project \
+# "
 
-WEBURL=http://localhost:8080
+WEBURL=http://gerrit.localhost
 #DATABASE_TYPE=postgresql
 AUTH_TYPE=LDAP
 LDAP_SERVER=ldap://ldap
@@ -28,3 +29,7 @@ LDAP_EMAILADDRESS=${email}
 LDAP_GROUPBASE=ou=Groups,dc=example,dc=org
 LDAP_GROUPMEMBERPATTERN=(member=${dn})
 LDAP_LOCALUSERNAMETOLOWERCASE=true
+
+# nginx-proxy
+VIRTUAL_HOST=gerrit.localhost
+VIRTUAL_PORT=8080
index 52a8e73..4f76cf1 100644 (file)
@@ -13,3 +13,7 @@ JENKINS_OPTS=""
 
 JENKINS_ADMIN_USER=admin
 JENKINS_ADMIN_PASSWORD=password
+
+# nginx-proxy
+VIRTUAL_HOST=jenkins.localhost
+VIRTUAL_PORT=8080
index 5d8fd1c..2e54adb 100644 (file)
@@ -26,7 +26,7 @@ objectClass: inetOrgPerson
 cn: Sandbox
 sn: User
 displayname: Sandbox User
-mail: sandbox@localhost
+mail: sandbox@example.org
 userpassword: {SSHA}WyQT/fg9FHJ/zcXsdvs51tk5vSlyL0fM
 
 dn: cn=sandbox-admins,ou=Groups,dc=example,dc=org
diff --git a/config/nginx/gerrit.conf b/config/nginx/gerrit.conf
deleted file mode 100644 (file)
index 2bd1ecb..0000000
+++ /dev/null
@@ -1,14 +0,0 @@
-server {
-  listen 8000;
-  server_name localhost;
-
-  location / {
-      proxy_set_header    Host $host:8000;
-      proxy_set_header    X-Real-IP $remote_addr;
-      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
-      proxy_set_header    X-Forwarded-Proto $scheme;
-      proxy_pass          http://gerrit:8080;
-      proxy_read_timeout  90;
-      proxy_redirect      http://gerrit:8080 http://localhost:8000;
-    }
-}
diff --git a/config/nginx/jenkins.conf b/config/nginx/jenkins.conf
deleted file mode 100644 (file)
index e87019c..0000000
+++ /dev/null
@@ -1,14 +0,0 @@
-server {
-  listen 8001;
-  server_name localhost;
-
-  location / {
-      proxy_set_header    Host $host:8001;
-      proxy_set_header    X-Real-IP $remote_addr;
-      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
-      proxy_set_header    X-Forwarded-Proto $scheme;
-      proxy_pass          http://jenkins:8080;
-      proxy_read_timeout  90;
-      proxy_redirect      http://jenkins:8080 http://localhost:8001;
-    }
-}
index 2440f48..b493244 100644 (file)
@@ -1,6 +1,15 @@
 ---
 version: '3'
 services:
+    init:
+        build: ./init
+        container_name: releng-init
+        volumes:
+            - init:/init/
+            - jenkins:/jenkins
+        links:
+            - jenkins
+            - gerrit
     ldap:
         image: osixia/openldap:latest
         container_name: releng-ldap
@@ -24,14 +33,23 @@ services:
         links:
             - ldap
             - nginx
+            - jenkins-agent
+    jenkins-agent:
+        build: ./jenkins/agent
+        container_name: releng-agent
+        ports:
+            - "22"
+        environment:
+            - JENKINS_SLAVE_SSH_PUBKEY=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUX11sDBXacCE/LBdcXO4E27OZbWtjNadXPGIRTN1leUFWJlnljlZT31Wbml8fvp+5BKbVRHP/W9IWj/PlBTxrxOPMN2Rch40tarPU1PxWJYM203n6Ac+GWKcfSPiikmfXiJ+pJHs+0E1MVhpLe08pb3qVdvwAGdQXVC53dg9ebrb5KirbLvWdEMewfVlxdB2ru2p358QN5Y7HHsAqW1sMr18G1/O5bVJl7g1JbnQ+/YjVaJRo7cDuonTOeatlEMfZZXzlermgXCcVFSIo/oKuKbtilZ2ye17ax9kN4rfVAAXIeOAUZWeCad32EdQEO7xL956ZftXyV+jlw03CndD5
     gerrit:
         image: openfrontier/gerrit:latest
         container_name: releng-gerrit
         env_file: config/gerrit.env
         volumes:
             - gerrit:/var/gerrit/review_site
+        expose:
+            - "8080"
         ports:
-            - "8081:8080"
             - "29418:29418"
         depends_on:
             - ldap
@@ -45,14 +63,14 @@ services:
         expose:
             - "5432"
     nginx:
-        image: nginx:latest
+        image: jwilder/nginx-proxy:latest
         restart: always
         container_name: releng-ingress
         ports:
-            - "8000:8000"
-            - "8001:8001"
+            - "80:80"
         volumes:
-            - ./config/nginx:/etc/nginx/conf.d
+            - /var/run/docker.sock:/tmp/docker.sock:ro
 volumes:
     jenkins:
     gerrit:
+    init:
diff --git a/init/Dockerfile b/init/Dockerfile
new file mode 100644 (file)
index 0000000..437a759
--- /dev/null
@@ -0,0 +1,31 @@
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2017 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+
+FROM ubuntu:14.04
+
+RUN DEBIAN_FRONTEND=noninteractive \
+    apt-get update -qq \
+    && apt-get install --no-install-recommends -y \
+       curl git python python-pip python-dev libyaml-dev \
+       realpath openssh-client \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir /docker-entrypoint-init.d/
+
+WORKDIR /docker-entrypoint-init.d/
+
+RUN /usr/bin/curl -sSL -O https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh
+RUN /bin/chmod +x wait-for-it.sh
+
+COPY config-ci-environment.sh /docker-entrypoint-init.d/
+
+RUN chmod +x /docker-entrypoint-init.d/*.sh
+
+CMD ["/docker-entrypoint-init.d/config-ci-environment.sh"]
diff --git a/init/config-ci-environment.sh b/init/config-ci-environment.sh
new file mode 100644 (file)
index 0000000..cf6fa4c
--- /dev/null
@@ -0,0 +1,180 @@
+#!/bin/bash -ex
+
+#
+# Each step is idempotent by creating a 'step-#.done' file after
+# successfully executing.
+#
+
+GERRIT_KEY=/init/id_rsa-sandbox
+JENKINS_KEY=/jenkins/.ssh/id_rsa
+SSH_OPTIONS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
+CI_MANAGEMENT_REPO=/init/ci-management
+GLOBAL_JJB_VERSION=v0.6.0
+
+# Generate a key for the sandbox user
+if [ ! -f /init/ssh-key-sandbox.done ]; then
+ssh-keygen -t rsa -N '' -f $GERRIT_KEY
+touch /init/ssh-key-sandbox.done
+fi
+
+##
+# Jenkins Setup
+##
+./wait-for-it.sh jenkins:8080 -t 30
+
+# Generate a key for the jenkins user
+if [ ! -f /init/ssh-key-jenkins.done ]; then
+mkdir -p /jenkins/.ssh/
+ssh-keygen -t rsa -N '' -f $JENKINS_KEY
+chown -R 1000:1000 /jenkins/.ssh/
+touch /init/ssh-key-jenkins.done
+fi
+
+##
+# Gerrit Login
+##
+./wait-for-it.sh gerrit:8080 -t 90
+
+# Be the first to login to gain Administrative rights
+if [ ! -f /init/step-1.done ]; then
+curl -X POST --data "username=sandbox&password=sandbox" http://gerrit:8080/login \
+    && touch /init/step-1.done
+fi
+
+##
+# Gerrit Setup
+##
+./wait-for-it.sh gerrit:29418 -t 90
+
+# Add generated ssh-pubkey to Gerrit keypairs
+if [ ! -f /init/step-2.done ]; then
+curl -X POST --user "sandbox:sandbox" -H "Content-type: plain/text" \
+    --data @"$GERRIT_KEY.pub" "http://gerrit:8080/a/accounts/self/sshkeys" \
+    && touch /init/step-2.done
+fi
+
+# Create Jenkins ssh user in Gerrit
+if [ ! -f /init/step-3.done ]; then
+ssh $SSH_OPTIONS -p 29418 sandbox@gerrit -i $GERRIT_KEY \
+    gerrit create-account jenkins-workshop --full-name "Jenkins\ Workshop" \
+    --group "Non-Interactive\ Users" --ssh-key - < "$JENKINS_KEY.pub" \
+    && touch /init/step-3.done
+fi
+
+# Create ci-management repository
+if [ ! -f /init/step-4.done ]; then
+ssh $SSH_OPTIONS -p 29418 sandbox@gerrit -i $GERRIT_KEY \
+    gerrit create-project ci-management --id --so --empty-commit \
+    -d "Workshop\ CI-Management\ Repo" -p "All-Projects" \
+    && touch /init/step-4.done
+fi
+
+# Populate ci-management repository with global-jjb
+if [ ! -f /init/step-5.done ]; then
+    ssh-keyscan -p 29418 gerrit >> /etc/ssh/ssh_known_hosts
+    git config --file ~/.gitconfig user.email "sandbox@example.org"
+    git config --file ~/.gitconfig user.name "sandbox"
+    eval "$(ssh-agent)"
+    ssh-add $GERRIT_KEY
+    git clone ssh://sandbox@gerrit:29418/ci-management.git $CI_MANAGEMENT_REPO
+    mkdir -p $CI_MANAGEMENT_REPO/jjb
+    cd $CI_MANAGEMENT_REPO/jjb
+    git submodule add https://gerrit.linuxfoundation.org/infra/releng/global-jjb
+    cd $CI_MANAGEMENT_REPO/jjb/global-jjb
+    git checkout $GLOBAL_JJB_VERSION
+    cd $CI_MANAGEMENT_REPO
+    git add jjb/global-jjb
+    git commit -am "Install global-jjb $GLOBAL_JJB_VERSION"
+    git push origin HEAD:refs/heads/master
+    touch /init/step-5.done
+fi
+
+# Populate ci-management with defaults
+if [ ! -f /init/step-6.done ]; then
+    cd $CI_MANAGEMENT_REPO
+    cat > $CI_MANAGEMENT_REPO/.gitreview <<-EOF
+[gerrit]
+host=gerrit.localhost
+port=29418
+username=sandbox
+project=ci-management.git
+defaultbranch=master
+EOF
+
+    cat > $CI_MANAGEMENT_REPO/jjb/ci-management.yaml <<-EOF
+---
+- project:
+    name: ci-jobs
+
+    jobs:
+      - '{project-name}-ci-jobs'
+
+    project: ci-management
+    project-name: ci-management
+    build-node: ciworkshop
+EOF
+
+    cat > $CI_MANAGEMENT_REPO/jjb/defaults.yaml <<-EOF
+---
+- defaults:
+    name: global
+
+    # lf-infra defaults
+    jenkins-ssh-credential: ciworkshop-jenkins-ssh
+    gerrit-server-name: ciworkshop
+    lftools-version: '<1.0.0'
+EOF
+    git add .
+    git commit -am "Initial JJB Files & gitreview"
+    git push origin HEAD:refs/heads/master
+    touch /init/step-6.done
+fi
+
+#  Upload Jenkins Jobs
+if [ ! -f /init/step-7.done ]; then
+    cd $CI_MANAGEMENT_REPO
+    pip install jenkins-job-builder
+    cat > $CI_MANAGEMENT_REPO/jenkins.ini <<-EOF
+[job_builder]
+ignore_cache=True
+keep_descriptions=False
+include_path=.:scripts:~/git/
+recursive=True
+
+[jenkins]
+url=http://jenkins:8080/
+user=sandbox
+password=sandbox
+query_plugins_info=True
+EOF
+    # Ensure JJB is installed first
+    jenkins-jobs --conf jenkins.ini update -r jjb/
+    touch /init/step-7.done
+fi
+
+# Add Verified Label
+if [ ! -f /init/step-8.done ]; then
+    eval "$(ssh-agent)"
+    ssh-add $GERRIT_KEY
+
+    ALL_PROJECTS=/tmp/All-Projects
+    mkdir -p /tmp/All-Projects
+
+    cd $ALL_PROJECTS
+    git init
+    git remote add origin ssh://sandbox@gerrit:29418/All-Projects.git
+    git fetch origin refs/meta/config:refs/remotes/origin/meta/config
+    git checkout meta/config
+
+    git config -f project.config label.Verified.function MaxWithBlock
+    git config -f project.config --add label.Verified.defaultValue 0
+    git config -f project.config --add label.Verified.value "-1 Fails"
+    git config -f project.config --add label.Verified.value "0 No score"
+    git config -f project.config --add label.Verified.value "+1 Verified"
+    git config -f project.config --add access.refs/heads/*.label-Verified "-1..+1 group Non-Interactive Users"
+
+    git commit -am "Create Verified Label"
+    git push origin meta/config:meta/config
+
+    touch /init/step-8.done
+fi
index 8c5aa83..2f8e6a3 100644 (file)
@@ -16,16 +16,23 @@ RUN echo 2.0 > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state
 COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
 RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
 
-# Set the instance hostname
-COPY hostname.groovy /usr/share/jenkins/ref/init.groovy.d/hostname.groovy
+ENV JENKINS_GROOVY_INIT /usr/share/jenkins/ref/init.groovy.d/
 
-# Setup basic security
-# TODO: Fix matrix permission causing NullPointerException ?
-# COPY authorization.groovy /usr/share/jenkins/ref/init.groovy.d/authorization.groovy
+# Configure Basic Security
+COPY basic-security-setup.groovy $JENKINS_GROOVY_INIT
 
-COPY ldap.groovy /usr/share/jenkins/ref/init.groovy.d/ldap.groovy
+# Configure LDAP
+COPY ldap.groovy $JENKINS_GROOVY_INIT
+
+# Configure Gerrit-Trigger
+COPY gerrit-config.groovy $JENKINS_GROOVY_INIT
+
+# Set Global Properties
+COPY global-properties.groovy $JENKINS_GROOVY_INIT
+
+# Create Jenkins SSH Agent
+COPY create-jenkins-agent.groovy $JENKINS_GROOVY_INIT
+
+# Create JJB INI in Config File Plugin
+COPY create-jjbini.groovy $JENKINS_GROOVY_INIT
 
-# TODO: Prevent Cross Site Script Forgery
-# TODO: Disable CLI over remoting
-# TODO: Enable Agent/Master Access Control
-# TODO: Disable JNLP Port
diff --git a/jenkins/agent/Dockerfile b/jenkins/agent/Dockerfile
new file mode 100644 (file)
index 0000000..bb61afc
--- /dev/null
@@ -0,0 +1,9 @@
+FROM jenkinsci/ssh-slave
+
+RUN DEBIAN_FRONTEND=noninteractive \
+    apt-get update \
+    && apt-get install --no-install-recommends -y python python-pip \
+       python-tox \
+    && apt-get clean
+
+RUN /usr/bin/pip install virtualenv
diff --git a/jenkins/authorization.groovy b/jenkins/authorization.groovy
deleted file mode 100644 (file)
index f716911..0000000
+++ /dev/null
@@ -1,42 +0,0 @@
-/*
- * SPDX-License-Identifier: EPL-1.0
- *
- * Copyright (c) 2017 The Linux Foundation and others.
- *
- * All rights reserved. This program and the accompanying materials
- * are made available under the terms of the Eclipse Public License v1.0
- * which accompanies this distribution, and is available at
- * http://www.eclipse.org/legal/epl-v10.html
- *
- */
-import jenkins.*
-import hudson.*
-import hudson.model.*
-import jenkins.model.*
-import hudson.security.*
-
-// Gets Admin username and password from environment
-//  and sets authentication to Jenkins' own user database,
-//  and the authorization to Matrix-based security.
-def hudsonRealm = new HudsonPrivateSecurityRealm(false)
-def admin_user = System.getenv('JENKINS_ADMIN_USER')
-def admin_password = System.getenv('JENKINS_ADMIN_PASSWORD')
-hudsonRealm.createAccount(admin_user, admin_password)
-
-def instance = Jenkins.getInstance()
-instance.setSecurityRealm(hudsonRealm)
-instance.save()
-
-def strategy = new GlobalMatrixAuthorizationStrategy()
-
-// Set Anonymous Permissions
-strategy.add(Jenkins.READ,'anonymous')
-strategy.add(Item.READ,'anonymous')
-strategy.add(Item.WORKSPACE,'anonymous')
-strategy.add(View.READ,'anonymous')
-
-// Set Admin Permissions
-strategy.add(Jenkins.ADMINISTER, "admin")
-
-instance.setAuthorizationStrategy(strategy)
-instance.save()
diff --git a/jenkins/basic-security-setup.groovy b/jenkins/basic-security-setup.groovy
new file mode 100644 (file)
index 0000000..e27f9ad
--- /dev/null
@@ -0,0 +1,33 @@
+/*
+ * SPDX-License-Identifier: EPL-1.0
+ *
+ * Copyright (c) 2017 The Linux Foundation and others.
+ *
+ * All rights reserved. This program and the accompanying materials
+ * are made available under the terms of the Eclipse Public License v1.0
+ * which accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ *
+ */
+import jenkins.model.Jenkins
+import jenkins.model.JenkinsLocationConfiguration
+import jenkins.security.s2m.AdminWhitelistRule
+import hudson.security.csrf.DefaultCrumbIssuer
+
+def instance = Jenkins.getInstance();
+
+// Enable Crumb issuer for CSRF protection
+instance.setCrumbIssuer(new DefaultCrumbIssuer(true))
+
+// Disable CLI Over Remoting
+instance.getDescriptor("jenkins.CLI").get().setEnabled(false)
+
+// Enable Agent -> Master subsystem
+instance.getInjector().getInstance(AdminWhitelistRule.class).setMasterKillSwitch(false)
+
+// Set the default URL
+def jlc = JenkinsLocationConfiguration.get()
+jlc.setUrl("http://jenkins.localhost/")
+jlc.save()
+
+instance.save()
diff --git a/jenkins/create-jenkins-agent.groovy b/jenkins/create-jenkins-agent.groovy
new file mode 100644 (file)
index 0000000..207cebc
--- /dev/null
@@ -0,0 +1,106 @@
+/*
+ * SPDX-License-Identifier: EPL-1.0
+ *
+ * Copyright (c) 2017 The Linux Foundation and others.
+ *
+ * All rights reserved. This program and the accompanying materials
+ * are made available under the terms of the Eclipse Public License v1.0
+ * which accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ *
+ */
+// TODO: There is a race condition when having the creation of an agent
+// and credentials in seperate scripts. The script to create an agent
+// will run before the credentials exist and won't be able to use them,
+// resulting in the need to restart jenkins container for them to be
+// picked up. There may just be a 'save()' call or something similar
+// missing, but this is the simplest approach to running the
+// sequentially.
+import com.cloudbees.jenkins.plugins.sshcredentials.impl.*
+import com.cloudbees.plugins.credentials.*
+import com.cloudbees.plugins.credentials.common.*
+import com.cloudbees.plugins.credentials.domains.*
+import com.cloudbees.plugins.credentials.impl.*
+import hudson.plugins.sshslaves.SSHLauncher;
+import hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy;
+import hudson.model.*
+import hudson.slaves.*
+import jenkins.model.*
+
+def global_domain = Domain.global()
+def store = Jenkins.getInstance().getExtensionList('com.cloudbees.plugins.credentials.SystemCredentialsProvider')[0].getStore()
+
+// This key should NEVER be used elsewhere. It is hardcoded only because the
+// publickey must be known ahead of time and passed as a variable to the docker
+// container.
+def    private_key = new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(
+"""-----BEGIN RSA PRIVATE KEY-----
+MIIEowIBAAKCAQEA1F9dbAwV2nAhPywXXFzuBNuzmW1rYzWnVzxiEUzdZXlBViZZ
+5Y5WU99Vm5pfH76fuQSm1URz/1vSFo/z5QU8a8TjzDdkXIeNLWqz1NT8ViWDNtN5
++gHPhlinH0j4opJn14ifqSR7PtBNTFYaS3tPKW96lXb8ABnUF1Qud3YPXm62+Soq
+2y71nRDHsH1ZcXQdq7tqd+fEDeWOxx7AKltbDK9fBtfzuW1SZe4NSW50Pv2I1WiU
+aO3A7qJ0znmrZRDH2WV85Xq5oFwnFRUiKP6Crim7YpWdsnte2sfZDeK31QAFyHjg
+FGVngmnd9hHUBDu8S/eemX7V8lfo5cNNwp3Q+QIDAQABAoIBAAZ+dAjdxb1MOHgK
+DRzR6qVTYoaKhgIeneNZAVauFwcHUiwkOBOA6rrd1WxQqB/8YD30GnXjBfkFAcOW
+20phgpt5Bc4002jQ7Ew7OwyDBsRLmVuP0+cFLydYhqO6Q4AVIf/BOcCeUPZ2wCZZ
+a6xrNNx7gDAZ11LZd1bPSCx2+7lTcxk6QmGmHic4xWbCDJ+/jDdzirMj083VxeVg
+JH26mbzCk/NQ4uhQqifxXkv8wCCpU1AdDl21sFDEJW2tgSoEruNN52TavXffH8f4
+mnyzFV2MRI1R3wwXLZwEW/jHWQFruIU0LSkS9eNZORog+cuoi5N9IDiBdfebdRsl
+gHvHQAECgYEA7wXML5phe/+t0xZqah4S/OS5igO/URhQ4gl+1rDH0EPQx6mM3l5Z
+qe9cV6tq9/sFjbK4UX/1np45NQ9i+olpvTLwiDbBczW3ySjgkQpUpulO2AD5jJiX
+aRlrbwN9WuL3T2P2WHRscvDVV0AMlVshOx09YSvDmoPjxtoUFP9iRgECgYEA43T7
+aaLYveorsLOxasC07tRS2PUc1NrO9etaLjfGtUcKcdIzS12vs6CHtvocgjeFA6No
+aF58ushuluQD7r9QD0Ao/IF8LAXnWXB/4Kp4inFbYuxJPdEMHC51dB1jmiomGepM
+EGhCqzQvS4U2Tu5MKtKpai+wL4ri7JI/4DMruvkCgYB6nerFcNkZl2xAoXstvQfY
+nC1iU9HNdD/p9R0QXdfjSybLhnsxiA1PU+93OgTB+hA7RLexd4c1O831HlOUWvHX
+kU47Unui8qe5ljK9tSMADSfZP4bFTXI/BD9Mz+l6unxMSeeSMQeBX3LSM5VA+WLu
+xG08cAsENSygUjeDHg/4AQKBgQDcoxlNux2r+38uBODQwOXB1kwXEI1LHIUtn4L2
+2jvylFrZViFTtik9gTakk7Ebz2dDxDr/IsizFsHPtJbr/MBYStB3P9OHkKJ969bf
+w/zxrkwLhVD2mdW5cIeWfvujC8ex08i9EaW6FQDbrPilUBqqX3be/itVss+005kK
+jhiZYQKBgCC9Oe3im2PFfQjm1yRFeTRNJ+f31sk78aLEJk+KF6yv2dfDgC1ChU6a
+QxR/mcboAaX1EvMLTjeFqDC5XK6gwC67s0H0p2gC91e6hzQs5Qb7M7g7LqD28YEW
+Tja198bt428xTsjxxffuhekDQt8hbEO6RJFJPxhHQIWVo8708lWj
+-----END RSA PRIVATE KEY-----""")
+
+def private_key_on_master = new BasicSSHUserPrivateKey.UsersPrivateKeySource();
+
+// Credentials for connecting to an SSH Jenkins Agent
+def jenkins_agent_credentials = new BasicSSHUserPrivateKey(
+  CredentialsScope.GLOBAL,
+  "jenkins-ssh-key",
+  "jenkins",
+  private_key,
+  null, // password
+  null // description
+)
+
+// ID must match 'jenkins-ssh-credentials in Global-JJB
+// Username must match Gerrit ssh user
+def gerrit_credentials = new BasicSSHUserPrivateKey(
+  CredentialsScope.GLOBAL,
+  "ciworkshop-jenkins-ssh", // ID
+  "jenkins-workshop", // username
+  private_key_on_master,
+  null, // password
+  "Gerrit User" // description
+)
+
+store.addCredentials(global_domain, jenkins_agent_credentials)
+store.addCredentials(global_domain, gerrit_credentials)
+
+// Create Jenkins SSH Agent without verifying host key.
+def ssh_strategy = new NonVerifyingKeyVerificationStrategy()
+def ssh_launcher = new SSHLauncher(
+    "jenkins-agent", 22, "jenkins-ssh-key",
+    null, null,
+    null, null,
+    60, 10, 10,
+    ssh_strategy
+)
+
+Slave slave = new DumbSlave("jenkins-agent1", "/home/jenkins", ssh_launcher)
+slave.setNodeDescription("CI Workshop Jenkins SSH Agent")
+slave.setRetentionStrategy(new RetentionStrategy.Always())
+slave.setLabelString("ciworkshop")
+
+Jenkins.getInstance().addNode(slave)
diff --git a/jenkins/create-jjbini.groovy b/jenkins/create-jjbini.groovy
new file mode 100644 (file)
index 0000000..b1f8f11
--- /dev/null
@@ -0,0 +1,34 @@
+/*
+ * SPDX-License-Identifier: EPL-1.0
+ *
+ * Copyright (c) 2017 The Linux Foundation and others.
+ *
+ * All rights reserved. This program and the accompanying materials
+ * are made available under the terms of the Eclipse Public License v1.0
+ * which accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ *
+ */
+import org.jenkinsci.plugins.configfiles.GlobalConfigFiles;
+import org.jenkinsci.plugins.configfiles.custom.CustomConfig;
+import jenkins.model.Jenkins;
+
+// Need to wait until after plugins are loaded
+Thread.start {
+    sleep 10000
+    println "--> Creating the JJB INI"
+
+    GlobalConfigFiles store = Jenkins.getInstance().getExtensionList(GlobalConfigFiles.class).get(GlobalConfigFiles.class);
+    CustomConfig config = new CustomConfig("jjbini", "jenkins-jjb-ini", "Jenkins Job Builder Config","""[job_builder]
+ignore_cache=True
+keep_descriptions=False
+include_path=.:scripts:~/git/
+recursive=True
+
+[jenkins]
+url=http://jenkins:8080/
+user=sandbox
+password=sandbox
+query_plugins_info=True""");
+    store.save(config);
+}
diff --git a/jenkins/gerrit-config.groovy b/jenkins/gerrit-config.groovy
new file mode 100644 (file)
index 0000000..73a491a
--- /dev/null
@@ -0,0 +1,59 @@
+/*
+ * SPDX-License-Identifier: EPL-1.0
+ *
+ * Copyright (c) 2017 The Linux Foundation and others.
+ *
+ * All rights reserved. This program and the accompanying materials
+ * are made available under the terms of the Eclipse Public License v1.0
+ * which accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ *
+ */
+import com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer;
+import com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritConnectionListener;
+import com.sonyericsson.hudson.plugins.gerrit.trigger.PluginImpl;
+import com.sonyericsson.hudson.plugins.gerrit.trigger.config.Config;
+
+import net.sf.json.JSONObject;
+import net.sf.json.JSONSerializer;
+
+Thread.start {
+    sleep 10000
+    println "--> Configuring the CI Workshop Gerrit Server"
+    def GerritServerName = "ciworkshop"
+    def plugin = PluginImpl.getInstance()
+
+    GerritServer gerritServer = new GerritServer(GerritServerName);
+
+    def configJSONString = """{"gerritFrontEndUrl":"http://gerrit.localhost",
+    "gerritHostName":"gerrit",
+    "gerritSshPort":"29418",
+    "gerritUserName":"jenkins-workshop",
+    "verdictCategories":[
+    {"verdictValue":"Code-Review","verdictDescription":"Code Review"},
+    {"verdictValue":"Verified","verdictDescription":"Verified"}],
+    "gerritBuildStartedVerifiedValue":"0",
+    "gerritBuildSuccessfulVerifiedValue":"1",
+    "gerritBuildFailedVerifiedValue":"-1",
+    "gerritBuildUnstableVerifiedValue":"0",
+    "gerritBuildNotBuiltVerifiedValue":"0",
+    "gerritBuildStartedCodeReviewValue":"0",
+    "gerritBuildSuccessfulCodeReviewValue":"0",
+    "gerritBuildFailedCodeReviewValue":"0",
+    "gerritBuildUnstableCodeReviewValue":"-1",
+    "gerritBuildNotBuiltCodeReviewValue":"0"}"""
+    JSONObject configObject = (JSONObject)JSONSerializer.toJSON(configJSONString);
+    Config config = new Config(configObject);
+
+    gerritServer.setConfig(config);
+    gerritServer.addListener(new GerritConnectionListener(GerritServerName));
+
+    if (plugin.containsServer(GerritServerName)) {
+        plugin.removeServer(plugin.getServer(GerritServerName))
+    }
+    plugin.addServer(gerritServer)
+
+    gerritServer.start()
+    gerritServer.startConnection()
+    println "--> Configuring the CI Workshop Gerrit Server...done"
+}
diff --git a/jenkins/global-properties.groovy b/jenkins/global-properties.groovy
new file mode 100644 (file)
index 0000000..31a21c6
--- /dev/null
@@ -0,0 +1,32 @@
+/*
+ * SPDX-License-Identifier: EPL-1.0
+ *
+ * Copyright (c) 2017 The Linux Foundation and others.
+ *
+ * All rights reserved. This program and the accompanying materials
+ * are made available under the terms of the Eclipse Public License v1.0
+ * which accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ *
+ */
+import jenkins.model.*
+
+instance = Jenkins.getInstance()
+globalNodeProperties = instance.getGlobalNodeProperties()
+envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class)
+
+newEnvVarsNodeProperty = null
+envVars = null
+
+if ( envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0 ) {
+  newEnvVarsNodeProperty = new hudson.slaves.EnvironmentVariablesNodeProperty();
+  globalNodeProperties.add(newEnvVarsNodeProperty)
+  envVars = newEnvVarsNodeProperty.getEnvVars()
+} else {
+  envVars = envVarsNodePropertyList.get(0).getEnvVars()
+
+}
+
+envVars.put("GIT_URL", "ssh://gerrit:29418/")
+
+instance.save()
diff --git a/jenkins/hostname.groovy b/jenkins/hostname.groovy
deleted file mode 100644 (file)
index 519aa33..0000000
+++ /dev/null
@@ -1,16 +0,0 @@
-/*
- * SPDX-License-Identifier: EPL-1.0
- *
- * Copyright (c) 2017 The Linux Foundation and others.
- *
- * All rights reserved. This program and the accompanying materials
- * are made available under the terms of the Eclipse Public License v1.0
- * which accompanies this distribution, and is available at
- * http://www.eclipse.org/legal/epl-v10.html
- *
- */
-import jenkins.model.JenkinsLocationConfiguration
-
-def jlc = JenkinsLocationConfiguration.get()
-jlc.setUrl("http://localhost:8001/")
-jlc.save()
index 1908b67..823bc3f 100644 (file)
@@ -33,10 +33,10 @@ def strategy = new GlobalMatrixAuthorizationStrategy()
 
 strategy.add(Jenkins.ADMINISTER, 'sandbox-admins')
 
-strategy.add(hudson.model.Hudson.READ,'anonymous')
-strategy.add(hudson.model.Item.READ,'anonymous')
-strategy.add(hudson.model.Item.WORKSPACE,'anonymous')
-strategy.add(hudson.model.View.READ,'anonymous')
+strategy.add(Jenkins.READ,'anonymous')
+strategy.add(Item.READ,'anonymous')
+strategy.add(Item.WORKSPACE,'anonymous')
+strategy.add(View.READ,'anonymous')
 
 instance.setAuthorizationStrategy(strategy)
 
index 931d79a..e741ab6 100644 (file)
@@ -1,71 +1,76 @@
-plain-credentials:1.4
-token-macro:2.1
-junit:1.20
-cloudbees-folder:6.1.0
-github:1.27.0
-handlebars:1.1.1
-pam-auth:1.3
-ssh-credentials:1.13
-workflow-scm-step:2.6
-github-api:1.86
-build-timeout:1.18
-branch-api:2.0.11
+ace-editor:1.1
 ant:1.5
-pipeline-input-step:2.7
-pipeline-milestone-step:1.3.1
-pipeline-model-definition:1.1.9
+antisamy-markup-formatter:1.5
+authentication-tokens:1.3
+bouncycastle-api:2.16.2
+branch-api:2.0.11
+build-timeout:1.18
+cloudbees-folder:6.1.0
+config-file-provider:2.16.3
+credentials-binding:1.12
+credentials:2.1.14
+display-url-api:2.0
+docker-commons:1.8
+docker-workflow:1.12
+durable-task:1.14
 email-ext:2.58
+external-monitor-job:1.7
+gerrit-trigger:2.25.0
+git-client:2.4.6
+git-server:1.7
 git:3.4.1
-docker-commons:1.8
-workflow-support:2.14
-scm-api:2.2.0
-pipeline-build-step:2.5.1
-workflow-durable-task-step:2.13
-pipeline-stage-view:2.8
-momentjs:1.1.1
+github-api:1.86
+github-branch-source:2.2.2
+github:1.27.0
 gradle:1.27.1
-authentication-tokens:1.3
-git-server:1.7
-workflow-multibranch:2.16
-pipeline-stage-tags-metadata:1.1.9
-workflow-basic-steps:2.6
+handlebars:1.1.1
+icon-shim:2.0.3
+jackson2-api:2.7.3
+jclouds-jenkins:2.14
 jquery-detached:1.2.1
-structs:1.9
-docker-workflow:1.12
-workflow-job:2.12.1
+junit:1.20
+ldap:1.16
 mailer:1.20
-workflow-cps:2.37
-pipeline-model-extensions:1.1.9
-jackson2-api:2.7.3
-resource-disposer:0.6
-external-monitor-job:1.7
-pipeline-model-api:1.1.9
-workflow-step-api:2.12
-timestamper:1.8.8
-credentials-binding:1.12
-workflow-api:2.19
-credentials:2.1.14
+mapdb-api:1.0.9.0
+mask-passwords:2.10.1
+matrix-auth:1.7
 matrix-project:1.11
-script-security:1.30
-subversion:2.9
+momentjs:1.1.1
+pam-auth:1.3
+pipeline-build-step:2.5.1
 pipeline-github-lib:1.0
-durable-task:1.14
-antisamy-markup-formatter:1.5
-github-branch-source:2.2.2
-pipeline-rest-api:2.8
-bouncycastle-api:2.16.2
-git-client:2.4.6
 pipeline-graph-analysis:1.4
-matrix-auth:1.7
-display-url-api:2.0
-ws-cleanup:0.34
-icon-shim:2.0.3
+pipeline-input-step:2.8
+pipeline-milestone-step:1.3.1
+pipeline-model-api:1.1.9
 pipeline-model-declarative-agent:1.1.1
-workflow-cps-global-lib:2.8
+pipeline-model-definition:1.1.9
+pipeline-model-extensions:1.1.9
+pipeline-rest-api:2.8
+pipeline-stage-step:2.2
+pipeline-stage-tags-metadata:1.1.9
+pipeline-stage-view:2.8
+plain-credentials:1.4
+resource-disposer:0.6
+scm-api:2.2.0
+script-security:1.31
+ssh-agent:1.15
+ssh-credentials:1.13
 ssh-slaves:1.20
-workflow-aggregator:2.5
-ace-editor:1.1
+structs:1.9
+subversion:2.9
+timestamper:1.8.8
+token-macro:2.1
 windows-slaves:1.3.1
-ldap:1.16
-pipeline-stage-step:2.2
-mapdb-api:1.0.9.0
+workflow-aggregator:2.5
+workflow-api:2.19
+workflow-basic-steps:2.6
+workflow-cps-global-lib:2.8
+workflow-cps:2.37
+workflow-durable-task-step:2.13
+workflow-job:2.12.1
+workflow-multibranch:2.16
+workflow-scm-step:2.6
+workflow-step-api:2.12
+workflow-support:2.14
+ws-cleanup:0.34