From 0a00894430dd4bd69567fe9bfcb3b18016e38543 Mon Sep 17 00:00:00 2001 From: Trevor Bramwell Date: Wed, 9 Aug 2017 10:47:35 -0700 Subject: [PATCH] Initial Commit Change-Id: I8cfcd9adc88146a32c7621470a4dcbf5b5643690 Signed-off-by: Trevor Bramwell --- README.rst | 141 ++++++++++++++++++++++++++++++++++++++ config/gerrit.env | 30 ++++++++ config/jenkins.env | 15 ++++ config/ldap.env | 25 +++++++ config/ldap/bootstrap/groups.ldif | 36 ++++++++++ config/nginx/gerrit.conf | 14 ++++ config/nginx/jenkins.conf | 14 ++++ config/postgres.env | 13 ++++ docker-compose.yml | 58 ++++++++++++++++ jenkins/Dockerfile | 31 +++++++++ jenkins/authorization.groovy | 42 ++++++++++++ jenkins/hostname.groovy | 16 +++++ jenkins/ldap.groovy | 43 ++++++++++++ jenkins/plugins.txt | 71 +++++++++++++++++++ 14 files changed, 549 insertions(+) create mode 100644 README.rst create mode 100644 config/gerrit.env create mode 100644 config/jenkins.env create mode 100644 config/ldap.env create mode 100644 config/ldap/bootstrap/groups.ldif create mode 100644 config/nginx/gerrit.conf create mode 100644 config/nginx/jenkins.conf create mode 100644 config/postgres.env create mode 100644 docker-compose.yml create mode 100644 jenkins/Dockerfile create mode 100644 jenkins/authorization.groovy create mode 100644 jenkins/hostname.groovy create mode 100644 jenkins/ldap.groovy create mode 100644 jenkins/plugins.txt diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..3c1dac5 --- /dev/null +++ b/README.rst @@ -0,0 +1,141 @@ +Linux Foundation Release Engineering Workshop +============================================= + +This is a container environment containing services used by the Linux +Foundation Release Engineering team. It is loosely influenced by the work +done by openfrontier_ (author of the Gerrit container). + +The primary services are: + + * Gerrit + * Jenkins + * Nexus [TODO] + +And the secondary services that support these: + + * OpenLDAP + * NGINX + * MariaDB [TODO] + * HAProxy [TODO] + +Most of the documentation for individual services and how they are +configured can be found under the `config/` directory, and +`$SERVICE.env` files contain environment_ variables used by the docker +containers. + +Quick Reference +--------------- + +Jenkins + localhost:8001 + +Gerrit + localhost:8000 + localhost:29418 + +Default user account: sandbox/sandbox + +Getting Started +--------------- + +.. code-block:: + + docker-compose up -d + +Will bring up an environment containing all the services with +authentication backed by LDAP, a simple ci-management repo in +Gerrit[TODO], and a basic job in Jenkins that verifies commits to the +ci-management repo[TODO]. + +To bring up a single service in the foreground you can use[TODO - Nginx]: + +.. code-block:: + + docker-compose up $SERVICE + +Note: dependent services will still be launched but in the background. +Hitting '^C' will stop this service, but not the others. + +If a service is backed by a Dockerfile (most will be eventually), then +changes to the Dockerfile or files under '$SERVICE/' will require +rebuilding the container: + +.. code-block:: + + docker-compose build + +To tear down the environment removing all the volumes, and start from +scratch, run: + +.. code-block:: + + docker-compose down -v + +For other useful docker-compose commands such as logs, see:: + + docker-compose -h + +Default User +------------ + +The default username and password in LDAP are: + + user: sandbox + pass: sandbox + +This user is part of the 'sandbox-admins' group and should have full +admin rights on all services. + +Goals +----- + +The goal of this project is to have an easily created workshop where +releng work can be tested or proof-of-concepts created. + +Some examples: + + * Jenkins Plugin upgrades + * Gerrit upgrades + * Jenkins and Gerrit-Trigger testing + * Nexus configuration + * Gerrit/LDAP group integration + +TODO +---- + +The following is a list of automation tasks still needed before the +environment can be considered usable: + +Nginx: +- [ ] Configure NGINX container to use environment variables_ + Note: If both Jenkins and Gerrit aren't both started, the NGINX + container will continuously restart. This can be worked around by + disabling the other service you don't want to use: + `mv nginx/config/gerrit.conf gerrit.conf.disabled` + +Nexus: +- [ ] Setup and configure Nexus + +Gerrit: +- [ ] Remove postgres container configuration and replace with MariaDB + (or make optional) + +General: +- [ ] Setup OpenLDAP over SSL by default +- [ ] Create a basic ci-management repository in Gerrit +- [ ] Connect and configure Gerrit and Jenkins automatically + - [ ] Have the Gerrit configuration setup in Jenkins + - [ ] Create Gerrit user with ssh pubkey from Jenkins + +Jenkins: +- [ ] Fix (on Jenkins restart):: + WARNING: Caught exception evaluating: + instance.hasExplicitPermission(attrs.sid,p) in /configureSecurity/. + Reason: java.lang.NullPointerException +- [ ] Create a basic SSH Jenkins agent that jobs can be ran on +- [ ] Disable CLI remoting +- [ ] Enable Agent -> Master Access Controls + +.. _environment: https://docs.docker.com/compose/environment-variables/#configuring-compose-using-environment-variables +.. _variables: https://docs.docker.com/samples/nginx/#using-environment-variables-in-nginx-configuration +.. _openfrontier: https://github.com/openfrontier/ci-compose diff --git a/config/gerrit.env b/config/gerrit.env new file mode 100644 index 0000000..e25e5a4 --- /dev/null +++ b/config/gerrit.env @@ -0,0 +1,30 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2017 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## + +#GERRIT_INIT_ARGS=" \ +# --install-plugin=commit-message-length-validator \ +# --install-plugin=replication \ +#" +# --install-plugin=delete-project \ + +WEBURL=http://localhost:8080 +#DATABASE_TYPE=postgresql +AUTH_TYPE=LDAP +LDAP_SERVER=ldap://ldap +LDAP_USERNAME=cn=readonly,dc=example,dc=org +LDAP_PASSWORD=readonly +LDAP_ACCOUNTBASE=ou=Users,dc=example,dc=org +LDAP_ACCOUNTPATTERN=(&(objectClass=inetOrgPerson)(uid=${username})) +LDAP_ACCOUNTFULLNAME=${cn} +LDAP_EMAILADDRESS=${email} +LDAP_GROUPBASE=ou=Groups,dc=example,dc=org +LDAP_GROUPMEMBERPATTERN=(member=${dn}) +LDAP_LOCALUSERNAMETOLOWERCASE=true diff --git a/config/jenkins.env b/config/jenkins.env new file mode 100644 index 0000000..52a8e73 --- /dev/null +++ b/config/jenkins.env @@ -0,0 +1,15 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2017 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +JAVA_OPTS="-Djenkins.install.runSetupWizard=false" +JENKINS_OPTS="" + +JENKINS_ADMIN_USER=admin +JENKINS_ADMIN_PASSWORD=password diff --git a/config/ldap.env b/config/ldap.env new file mode 100644 index 0000000..dbc43a6 --- /dev/null +++ b/config/ldap.env @@ -0,0 +1,25 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2017 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +LDAP_ORGANISATION=Test Org +LDAP_DOMAIN=example.org +# Only set this if you want it different from the LDAP_DOMAIN +LDAP_BASE_DN=dc=example,dc=org + +LDAP_READONLY_USER=true +LDAP_READONLY_USER_USERNAME=readonly +LDAP_READONLY_USER_PASSWORD=readonly + +LDAP_ADMIN_PASSWORD=admin + +LDAP_TLS=false +LDAP_TLS_ENFORCE=false + +LDAP_REMOVE_CONFIG_AFTER_SETUP=false diff --git a/config/ldap/bootstrap/groups.ldif b/config/ldap/bootstrap/groups.ldif new file mode 100644 index 0000000..5d8fd1c --- /dev/null +++ b/config/ldap/bootstrap/groups.ldif @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2017 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +dn: ou=Users,dc=example,dc=org +changetype: add +objectClass: organizationalUnit +ou: Users + +dn: ou=Groups,dc=example,dc=org +changetype: add +objectClass: organizationalUnit +ou: Groups + +# Password: sandbox +dn: uid=sandbox,ou=Users,dc=example,dc=org +changetype: add +objectClass: top +objectClass: person +objectClass: inetOrgPerson +cn: Sandbox +sn: User +displayname: Sandbox User +mail: sandbox@localhost +userpassword: {SSHA}WyQT/fg9FHJ/zcXsdvs51tk5vSlyL0fM + +dn: cn=sandbox-admins,ou=Groups,dc=example,dc=org +changetype: add +objectClass: top +objectClass: groupOfNames +member: uid=sandbox,ou=Users,dc=example,dc=org diff --git a/config/nginx/gerrit.conf b/config/nginx/gerrit.conf new file mode 100644 index 0000000..2bd1ecb --- /dev/null +++ b/config/nginx/gerrit.conf @@ -0,0 +1,14 @@ +server { + listen 8000; + server_name localhost; + + location / { + proxy_set_header Host $host:8000; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_pass http://gerrit:8080; + proxy_read_timeout 90; + proxy_redirect http://gerrit:8080 http://localhost:8000; + } +} diff --git a/config/nginx/jenkins.conf b/config/nginx/jenkins.conf new file mode 100644 index 0000000..e87019c --- /dev/null +++ b/config/nginx/jenkins.conf @@ -0,0 +1,14 @@ +server { + listen 8001; + server_name localhost; + + location / { + proxy_set_header Host $host:8001; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_pass http://jenkins:8080; + proxy_read_timeout 90; + proxy_redirect http://jenkins:8080 http://localhost:8001; + } +} diff --git a/config/postgres.env b/config/postgres.env new file mode 100644 index 0000000..bdffe8b --- /dev/null +++ b/config/postgres.env @@ -0,0 +1,13 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2017 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +POSTGRES_USER=gerrit2 +POSTGRES_PASSWORD=gerrit +POSTGRES_DB=reviewdb diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..2440f48 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,58 @@ +--- +version: '3' +services: + ldap: + image: osixia/openldap:latest + container_name: releng-ldap + env_file: config/ldap.env + command: "--loglevel debug --copy-service" + volumes: + - ./config/ldap/bootstrap/groups.ldif:/container/service/slapd/assets/config/bootstrap/ldif/custom/50-groups.ldif + ports: + - "636:636" + jenkins: + build: ./jenkins + container_name: releng-jenkins + env_file: config/jenkins.env + volumes: + - jenkins:/var/jenkins_home + expose: + - "8080" + depends_on: + - ldap + - nginx + links: + - ldap + - nginx + gerrit: + image: openfrontier/gerrit:latest + container_name: releng-gerrit + env_file: config/gerrit.env + volumes: + - gerrit:/var/gerrit/review_site + ports: + - "8081:8080" + - "29418:29418" + depends_on: + - ldap + - nginx + links: + - ldap + - nginx + postgres: + image: postgres:latest + env_file: config/postgres.env + expose: + - "5432" + nginx: + image: nginx:latest + restart: always + container_name: releng-ingress + ports: + - "8000:8000" + - "8001:8001" + volumes: + - ./config/nginx:/etc/nginx/conf.d +volumes: + jenkins: + gerrit: diff --git a/jenkins/Dockerfile b/jenkins/Dockerfile new file mode 100644 index 0000000..8c5aa83 --- /dev/null +++ b/jenkins/Dockerfile @@ -0,0 +1,31 @@ +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2017 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +FROM jenkins/jenkins:lts + +# Disable upgrade wizard +RUN echo 2.0 > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state + +# Install desired plugins +COPY plugins.txt /usr/share/jenkins/ref/plugins.txt +RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt + +# Set the instance hostname +COPY hostname.groovy /usr/share/jenkins/ref/init.groovy.d/hostname.groovy + +# Setup basic security +# TODO: Fix matrix permission causing NullPointerException ? +# COPY authorization.groovy /usr/share/jenkins/ref/init.groovy.d/authorization.groovy + +COPY ldap.groovy /usr/share/jenkins/ref/init.groovy.d/ldap.groovy + +# TODO: Prevent Cross Site Script Forgery +# TODO: Disable CLI over remoting +# TODO: Enable Agent/Master Access Control +# TODO: Disable JNLP Port diff --git a/jenkins/authorization.groovy b/jenkins/authorization.groovy new file mode 100644 index 0000000..f716911 --- /dev/null +++ b/jenkins/authorization.groovy @@ -0,0 +1,42 @@ +/* + * SPDX-License-Identifier: EPL-1.0 + * + * Copyright (c) 2017 The Linux Foundation and others. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + */ +import jenkins.* +import hudson.* +import hudson.model.* +import jenkins.model.* +import hudson.security.* + +// Gets Admin username and password from environment +// and sets authentication to Jenkins' own user database, +// and the authorization to Matrix-based security. +def hudsonRealm = new HudsonPrivateSecurityRealm(false) +def admin_user = System.getenv('JENKINS_ADMIN_USER') +def admin_password = System.getenv('JENKINS_ADMIN_PASSWORD') +hudsonRealm.createAccount(admin_user, admin_password) + +def instance = Jenkins.getInstance() +instance.setSecurityRealm(hudsonRealm) +instance.save() + +def strategy = new GlobalMatrixAuthorizationStrategy() + +// Set Anonymous Permissions +strategy.add(Jenkins.READ,'anonymous') +strategy.add(Item.READ,'anonymous') +strategy.add(Item.WORKSPACE,'anonymous') +strategy.add(View.READ,'anonymous') + +// Set Admin Permissions +strategy.add(Jenkins.ADMINISTER, "admin") + +instance.setAuthorizationStrategy(strategy) +instance.save() diff --git a/jenkins/hostname.groovy b/jenkins/hostname.groovy new file mode 100644 index 0000000..519aa33 --- /dev/null +++ b/jenkins/hostname.groovy @@ -0,0 +1,16 @@ +/* + * SPDX-License-Identifier: EPL-1.0 + * + * Copyright (c) 2017 The Linux Foundation and others. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + */ +import jenkins.model.JenkinsLocationConfiguration + +def jlc = JenkinsLocationConfiguration.get() +jlc.setUrl("http://localhost:8001/") +jlc.save() diff --git a/jenkins/ldap.groovy b/jenkins/ldap.groovy new file mode 100644 index 0000000..1908b67 --- /dev/null +++ b/jenkins/ldap.groovy @@ -0,0 +1,43 @@ +/* + * SPDX-License-Identifier: EPL-1.0 + * + * Copyright (c) 2017 The Linux Foundation and others. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + */ +import jenkins.* +import jenkins.model.* +import hudson.* +import hudson.model.* +import hudson.security.* + +def instance = Jenkins.getInstance() + +def LDAPRealm = new LDAPSecurityRealm("ldap://ldap", +"dc=example,dc=org", +"ou=Users", +"uid={0}", +"ou=Groups", +"cn=readonly,dc=example,dc=org", +"readonly", +false, +false +) +instance.setSecurityRealm(LDAPRealm) + +def strategy = new GlobalMatrixAuthorizationStrategy() + +strategy.add(Jenkins.ADMINISTER, 'sandbox-admins') + +strategy.add(hudson.model.Hudson.READ,'anonymous') +strategy.add(hudson.model.Item.READ,'anonymous') +strategy.add(hudson.model.Item.WORKSPACE,'anonymous') +strategy.add(hudson.model.View.READ,'anonymous') + +instance.setAuthorizationStrategy(strategy) + +instance.save() diff --git a/jenkins/plugins.txt b/jenkins/plugins.txt new file mode 100644 index 0000000..931d79a --- /dev/null +++ b/jenkins/plugins.txt @@ -0,0 +1,71 @@ +plain-credentials:1.4 +token-macro:2.1 +junit:1.20 +cloudbees-folder:6.1.0 +github:1.27.0 +handlebars:1.1.1 +pam-auth:1.3 +ssh-credentials:1.13 +workflow-scm-step:2.6 +github-api:1.86 +build-timeout:1.18 +branch-api:2.0.11 +ant:1.5 +pipeline-input-step:2.7 +pipeline-milestone-step:1.3.1 +pipeline-model-definition:1.1.9 +email-ext:2.58 +git:3.4.1 +docker-commons:1.8 +workflow-support:2.14 +scm-api:2.2.0 +pipeline-build-step:2.5.1 +workflow-durable-task-step:2.13 +pipeline-stage-view:2.8 +momentjs:1.1.1 +gradle:1.27.1 +authentication-tokens:1.3 +git-server:1.7 +workflow-multibranch:2.16 +pipeline-stage-tags-metadata:1.1.9 +workflow-basic-steps:2.6 +jquery-detached:1.2.1 +structs:1.9 +docker-workflow:1.12 +workflow-job:2.12.1 +mailer:1.20 +workflow-cps:2.37 +pipeline-model-extensions:1.1.9 +jackson2-api:2.7.3 +resource-disposer:0.6 +external-monitor-job:1.7 +pipeline-model-api:1.1.9 +workflow-step-api:2.12 +timestamper:1.8.8 +credentials-binding:1.12 +workflow-api:2.19 +credentials:2.1.14 +matrix-project:1.11 +script-security:1.30 +subversion:2.9 +pipeline-github-lib:1.0 +durable-task:1.14 +antisamy-markup-formatter:1.5 +github-branch-source:2.2.2 +pipeline-rest-api:2.8 +bouncycastle-api:2.16.2 +git-client:2.4.6 +pipeline-graph-analysis:1.4 +matrix-auth:1.7 +display-url-api:2.0 +ws-cleanup:0.34 +icon-shim:2.0.3 +pipeline-model-declarative-agent:1.1.1 +workflow-cps-global-lib:2.8 +ssh-slaves:1.20 +workflow-aggregator:2.5 +ace-editor:1.1 +windows-slaves:1.3.1 +ldap:1.16 +pipeline-stage-step:2.2 +mapdb-api:1.0.9.0 -- 2.16.6