We recently lost our hudson server due to a multiple disk failure in the RAID array storing our hudson configuration. [5 of the 15 disks died]
So I’ve been looking into a backup script that will allow us to keep a backup of the configuration. We use Maven for most of our builds, so the released artifacts are in our Maven Repository (which is hosted on two servers each with RAID arrays and using DRBD to mirror between the pair, with an rsync to a NAS in another cabinet and we are trying to get an rsych to an off-site storage going as well).
Works on *nixfind ~/.m2/repository -type d -name *-SNAPSHOT -exec rm -rvf {} ;By searching for the directories we should catch the -YYYYMMDD.HHMMSS format of snapshots also
Cool post I just found:http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/7
I am starting a series of things you should not do:
Seam has this handy annotation: @Synchronized which ensures that only a single thread may access the methods/fields of the component at the same time.
Often times it is easy to forget that SESSION scoped Seam components are automatically @Synchronized
Java has this (formerly) handy modifier synchronized which when applied to a method, ensures that the object’s implicit lock is held whenever the method is invoked.
I’ve been meaning to blog about getting transaction management working with OpenEjb and Jetty using jetty:run… it’s still an on-going story… but the following might get you going…First off, in your pom.xml you need to add the configuration for maven-jetty-plugin… we need to dance around the various activemq/activeio versions and ensure that we get the correct version of ant… <project xmlns=“http://maven.apache.org/POM/4.0.0" xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=“http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> 4.0.0 org.apache.openejb.examples jetty-openejb war 1.0-SNAPSHOT jetty-openejb Maven Webapp http://maven.apache.org junit junit 3.8.1 test ${project.artifactId} org.mortbay.jetty maven-jetty-plugin 6.1.22 org.apache.activemq activemq-core 4.1.1 commons-logging commons-logging commons-logging commons-logging-api org.apache.activemq activeio-core org.apache.activemq activemq-ra 4.1.1 commons-logging commons-logging commons-logging commons-logging-api org.apache.activemq activeio-core org.apache.activemq activeio-core 3.1.2 commons-logging commons-logging commons-logging commons-logging-api org.apache.openejb openejb-core 3.1.2 org.apache.activemq activemq-core org.apache.activemq activemq-ra org.apache.activemq activeio-core junit junit org.mortbay.jetty jsp-2.1-jetty 6.1.22 ant ant ${basedir}/src/main/jetty/jetty.xml Next we need to configure a src/main/jetty/jetty.xml to bind the UserTransaction instance into jetty… java.naming.factory.initial org.apache.openejb.client.LocalInitialContextFactory openejb:TransactionManager And presto-chango, now jetty has a transaction manager provided by openejb. (Note: if we don’t mind storing that in a jetty-env in /WEB-INF, you can put the same config in WEB-INF/jetty-env.xml) OK, so here are the issues:Reloading does not work (because org.apache.openejb.core.ivm.naming.IvmContext does not support the destroySubcontext(Context) methodWe are using jetty’s JNDI provider in the web-app and openejb’s JNDI provider for the EJBs… this is because When jetty binds names to JNDI (using org.mortbay.jetty.plus.naming.Resource or org.mortbay.jetty.plus.naming.Transaction) it binds the object to JNDIName and it also binds a NamingEnrtry for the object to __/JNDIName Unfortunately, openejb’s JNDI implementation seems to be somewhat strange in this regard… if we add the SystemProperties to jetty to have it use openejb’s JNDI implementation, e.g. add the following to /project/build/plugins/plugin[maven-jetty-plugin]/configuration/systemProperties java.naming.factory.initial org.apache.openejb.client.LocalInitialContextFactory Then when we bind /UserTransaction it gets bound to openejb:/UserTransaction but when we lookup /UserTransaction openejb looks up openejb:local//UserTransaction And that is just for starters… there seems to be a whole host of other JNDI strangeness between jetty’s side and openejb’s sideThe side effect of all this is that if you want resource refs to work correctly, you need to fish them out of openejb’s JNDI context and push them into jetty’s JNDI context In any case this is at least a start!
Review Board is quite nice… it has a handy program for posting reviews (postreview)… and you can integrate this into your subversion hook scripts quite nicely…
But what if you want to automate submitting reviews on only parts of your code base…
What I want is to be able to set a property on a folder and then any time a file is changed in that folder or it’s children, then a review will automatically be scheduled…
Here is my version of the holy grail, i.e.Authentication for Apache HTTPD against Active Directory.This is not the only way to skin this cat, for example you could also use Sander Marechal’s technique (which uses mod_authnz_ldap). However, my problem with Sander’s technique is that you need to have an account in Active Directory which you will use to bind to the LDAP server. That means that the accound password has to be stored in a plain-text file on the Apache server, and if the password expires everything breaks until you go fix the password.I want as near to zero maintenance as possible, running on CentOS 5.2 with minimal custom work - so that I don’t have to maintain it when it does break.I have previously configured Kerberos to authenicate against Active Directory, so my first attempt was with mod_auth_kerb. That worked… but very slowly… trying to an empty access Subversion repository took over 2 minutes. The problem being that successful authentication was not being cached.Then I tried using mod_perl and the Authen::Simple::ActiveDirectory reasoning that I could always hack caching once in perl… but getting that lot installed on a CentOS 5.2 from RPMs was an exercise in tears.So, anyway, I found out that Cyrus SASL supports two modes of LDAP authentication:The bind method uses the LDAP bind facility to verify the password. The bind method is not available when ldap_use_sasl is turned on. In that case saslauthd will use fastbind.‘bind’ is the default auth method. When ldap_use_sasl is enabled, ‘fastbind’ is the default.The custom method uses userPassword attribute to verify the password. Suppored hashes: crypt, md5, smd5, sha and ssha. Cleartext is supported as well.The fastbind method (when ’ldap_use_sasl: no’) does away with the search and an extra anonymous bind in auth_bind, but makes two assumptions: 1. Expanding the ldap_filter expression gives the user’s fully-qualified DN 2. There is no cost to staying bound as a named userSo bind is pretty much the same technique as that used by mod_authnz_ldap, and (as ActiveDirectory does not support - at my company at least - anonymous bind) is ruled out of the running.“fastbind” sounds exactly like what I want… ok and saslauthd can be configured to cache authentication, so none of the performance hit of mod_auth_kerb. All we need is a way to tie this into Apache.So, enter mod_authn_sasl which does just that.First, we need to create an RPM, so I did a quicksudo yum install httpd-devel rpmbuild mocksudoecho “%_topdir %(echo $HOME)/rpmbuild” > ~/.rpmmacrosmkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}I added my account to the mock group, logged out and in again for the group membership to be recognised and I was ready to go. This was my first experience playing with mock, but it was lovely, especially as CentOS has its own version.I pre-primed my mock environments for i386 and x86_64 as I need RPMs for both of these:mock -r centos-5-i386 initmock -r centos-5-x86_64 initThen I downloaded mod_authn_sasl-1.0.2.tar.bz2 and started writing my spec file.You can download the RPC spec: mod_authn_sasl.spec and the apache config file: mod_authn_sasl.conf.I put mod_authn_sasl-1.0.2.tar.bz2 and mod_authn_sasl.conf in the ~/rpmbuild/SOURCES and then runningrpmbuild -ba mod_authn_sasl.specproduced my source RPM, then I used mock to create the two binary RPMs that I needed:mock -r centos-5-i386 ~/rpmbuild/SRPMS/mod_authn_sasl-1.0.2-4.src.rpmmock -r centos-5-x86_64 ~/rpmbuild/SRPMS/mod_authn_sasl-1.0.2-4.src.rpmThe rpms built from the mock environments end up in /var/lib/mock/centos-5-i386/result and /var/lib/mock/centos-5-x86_64/result. If you want to be lazy and trust my binaries here they are:mod_authn_sasl-1.0.2-4.i386.rpmmod_authn_sasl-1.0.2-4.x86_64.rpmNow all we need to do is configure everything.I’ve written this shell script to simplify configuring saslauthd for most good deployments of Active Directory, i.e. where there are SRV records for the domain in your DNS server. You need to run it as root. It looks up the SRV records for the domain name you provide, and creates /etc/saslauthd.conf from them, assuming that the Active Directory domain name is the first word of your DNS name converted to uppercase. So for example, if your Active Directory is set up correctly, and you can login to windows as either EXAMPLE\joebloggs or jbloggs@example.foo.com then you would runsh system-saslauthd-active-directory-config.sh example.foo.comAnd that should setup saslauthd for you. You can test this using the testsaslauthd program.Then all you need to do is secure the appropriate locations in apache, e.g. by adding the following to your VirtualHost configuration:<Location /private>AuthSaslPwcheckMethod saslauthdAuthSaslAppname httpdAuthSaslRealm exampleAuthType basicAuthBasicProvider saslAuthBasicAuthoritative OnAuthName “sasl@example.com"require valid-userAnd define the SASL application provider for the AuthSaslAppname you specified, e.g. for the above configuration you need to create /usr/lib/sasl2/httpd.conf with the contents:pwcheck_method:saslauthdThat’s it. You should be done.Updated Friday 28th November 2008: I noticed that the spec file I had provided did not have the correct Requires and BuildRequires. I have fixed this and now I have posted updated rpms (these are 1.0.2-4) with the correct requires to help with a minimal install
Here’s my step by step:Install CentOS 5.2Configure Network and Proxies as neededI usually create a login script: /etc/profile.d/login.sh as follows:function set_proxies() { local s PROXY_ADDR=“http://proxy.example.com:8000/" for s in HTTP HTTPS FTP GOPHER NEWSPOST NEWSREPLY\ NEWS NNTP SNEWSPOST SNEWSREPLY SNEWS\ WAIS FINGER CSO; do export ${s}_PROXY=${PROXY_ADDR} done for s in http https ftp; do export ${s}_proxy=${PROXY_ADDR} done}set_proxiesI installed from the DVD, so yum update to ensure everything is up-to-date.Get my mod_authn_sasl rpm:wget http://www.one-dash.com/blog/mod_authn_sasl-1.0.2-3.i386.rpmInstall it (this should pull down the):yum –nogpgcheck localinstall mod_authn_sasl-1.0.2-3.i386.rpmGet my sasl-magic-config script:wget http://www.one-dash.com/blog/system-saslauthd-active-directory-config.shdos2unix system-saslauthd-active-directory-config.shRun the script:sh system-saslauthd-active-directory-config.sh example.comChange the security level:system-config-securitylevel-tuiSet SELinux to Permissive, Customize and enable WWW and Secure WWW on the firewallCreate the sasl2 configuration for apache:echo “pwcheck_method:saslauthd” > /usr/lib/sasl2/apache-httpd.confNow we need to install Subversion 1.5.2:wget http://summersoft.fay.ar.us/pub/subversion/1.5.2/rhel-5/i386/subversion-1.5.2-1.i386.rpmwget http://summersoft.fay.ar.us/pub/subversion/1.5.2/rhel-5/i386/neon-0.27.2-1.i386.rpmwget http://summersoft.fay.ar.us/pub/subversion/1.5.2/rhel-5/i386/mod_dav_svn-1.5.2-1.i386.rpmyum install perl-URIrpm -i neon-0.27.2-1.i386.rpm rpm -i subversion-1.5.2-1.i386.rpmrpm -i mod_dav_svn-1.5.2-1.i386.rpmNext modify the apache conf file for our subversion repositories: /etc/httpd/conf.d/subversion.conf# Needed to do Subversion Apache server.LoadModule dav_svn_module modules/mod_dav_svn.so# Only needed if you decide to do “per-directory” access control.#LoadModule authz_svn_module modules/mod_authz_svn.so<Location /svn> DAV svn SVNParentPath /var/www/svn # Limit write permission to list of valid users. # Require SSL connection for password protection. # SSLRequireSSL AuthType Basic AuthName “EXAMPLE” AuthBasicProvider sasl AuthSaslPwcheckMethod saslauthd auxprop AuthSaslAppname apache-httpd AuthSaslRealm example Require valid-user Next, create the subversion projects root and restart Apache:mkdir /var/www/svnchown -R apache.apache /var/www/svnservice httpd restartAt this point you can now create a test repository:svnadmin create /var/www/svn/testchown -R apache.apache /var/www/svn/testLet’s test if subversion is working:svn info http://localhost/svn/testsvn mkdir –username myWindowsUsername –message “a test commit” http://localhost/svn/test/trunkAt this point we should have subversion up and running.
One of my co-workers has asked me to post this up. It’s rough and ready, so make of it what you want.First off, Acer recently pushed an update for better performance with the Huawei USB modems… I’m assuming that you have this update… check if the file /etc/udev/rules.d/10-Huawei-Datacard.rules exists.If that file exists then when you plug in a E169g it will be correctly autodetected without requiring poking about with usbmodeswitch… it will bind the three serial ports of the E169G to /dev/HuaweiMobile-0, /dev/HuaweiMobile-1 and /dev/HuaweiMobile-2.OK, so assuming you see these device nodes after plugging in the E169G, the next problem is getting wvdial to connect. Here’s the wvdial.conf file I use[Dialer Defaults]Init2 = ATZInit3 = ATHInit4 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0Stupid Mode = 1Modem Type = USB ModemISDN = 0Phone = *99#Modem = /dev/HuaweiMobile-0Username = usernamePassword = passwordDial Command = ATDTBaud = 460800Init5 = AT+CGDCONT=1,“IP”,“3ireland.ie"By the way, the username is actually “username”, and the password is actually “password”.That should be enough to get any self-respecting linux freak 90% of the way there. There was some stuff I had to tweak to get DHCP to populate resolve.conf from the pppd connection… and I added the following udev rule as 70-huawei-e169g-dial.rulesSUBSYSTEM==“usb” SYSFS{idProduct}==“1001”,SYSFS{idVendor}==“12d1”,RUN+="/usr/sbin/e169g_dial"And, /usr/sbin/e169g_dial is just#!/bin/shsleep 5/usr/bin/wvdial 2>&1 > /var/log/wvdial &You might be able to tune down from 5 seconds if you can be bothered… but you need at least some delay
Life gets in the way… but we’re back with our final installment! So where to start, let’s start with a publisher for freestyle builds, then we’ll add a publisher for Maven 2 builds… These will both require some reports to display results, and then finally we’ll need the plugin entry point. But before we get into all that, perhaps I should briefly explain structured form submission supportDataBoundConstructorsHudson uses Stapler as it’s web framework. One of the things that Stapler provides is support for constructing objects from a JSON data model. Basically, if you have a class with a public constructor annotated with @DataBoundConstructor, Stapler will bind fields from a JSON object by matching the field name to the constructor parameter name. If a parameter also has a @DataBoundConstructor, then Stapler will recurse to construct this child object from the child JSON object.Note: The only hole in this (at the moment) is if you want to inject a variable class, i.e. it does not support the case where there are three ChildImpl classes all implementing Child, and all with @DataBoundConstructor and Parent’s constructor has a parameter which takes Child… However, plans are afoot to fix this!JavaNCSSPublisherPublishers in Hudson must have a Descriptor, this will be registered with Hudson and allows Hudson to create Publisher instances which have the details for the project they are publishing. Descriptors are normally implemented as an inner class called DescriptorImpl and there is normally a static field of the publisher DESCRIPTOR that holds the Descriptor singleton. 99.995% of the time, you will want your publisher to have a @DataBoundConstructor, so without further delay, here is the publisher:package hudson.plugins.javancss;import hudson.maven.MavenModule;import hudson.maven.MavenModuleSet;import hudson.model.AbstractProject;import hudson.model.Action; import hudson.model.Descriptor;import hudson.plugins.helpers.AbstractPublisherImpl;import hudson.plugins.helpers.Ghostwriter;import hudson.tasks.BuildStepDescriptor;import hudson.tasks.Publisher;import net.sf.json.JSONObject;import org.kohsuke.stapler.DataBoundConstructor;import org.kohsuke.stapler.StaplerRequest;public class JavaNCSSPublisher extends AbstractPublisherImpl { private String reportFilenamePattern; @DataBoundConstructor public JavaNCSSPublisher(String reportFilenamePattern) { reportFilenamePattern.getClass(); this.reportFilenamePattern = reportFilenamePattern; } public String getReportFilenamePattern() { return reportFilenamePattern; } public boolean needsToRunAfterFinalized() { return false; } public static final DescriptorImpl DESCRIPTOR = new DescriptorImpl(); public Descriptor getDescriptor() { return DESCRIPTOR; } public Action getProjectAction(AbstractProject project) { return new JavaNCSSProjectIndividualReport(project); } protected Ghostwriter newGhostwriter() { return new JavaNCSSGhostwriter(reportFilenamePattern); } public static final class DescriptorImpl extends BuildStepDescriptor { private DescriptorImpl() { super(JavaNCSSPublisher.class); } public String getDisplayName() { return “Publish " + PluginImpl.DISPLAY_NAME; } public boolean isApplicable(Class> extends AbstractBuildAction { private final Collection results; private final Statistic totals; public AbstractBuildReport(Collection results) { this.results = results; this.totals = Statistic.total(results); } public Collection getResults() { return results; } public Statistic getTotals() { return totals; } public String getSummary() { AbstractBuild prevBuild = getBuild().getPreviousBuild(); while (prevBuild != null && prevBuild.getAction(getClass()) == null) { prevBuild = prevBuild.getPreviousBuild(); } if (prevBuild == null) { return totals.toSummary(); } else { AbstractBuildReport action = prevBuild.getAction(getClass()); return totals.toSummary(action.getTotals()); } } public String getIconFileName() { return PluginImpl.ICON_FILE_NAME; } public String getDisplayName() { return PluginImpl.DISPLAY_NAME; } public String getUrlName() { return PluginImpl.URL; } public boolean isGraphActive() { return false; }}Similarly, we have AbstractProjectReport which will be used for project reports:package hudson.plugins.javancss;import java.io.IOException;import java.util.Collection;import java.util.Collections;import hudson.model.AbstractBuild;import hudson.model.AbstractProject;import hudson.model.ProminentProjectAction;import hudson.plugins.helpers.AbstractProjectAction;import hudson.plugins.javancss.parser.Statistic;import org.kohsuke.stapler.StaplerRequest;import org.kohsuke.stapler.StaplerResponse;public abstract class AbstractProjectReport<T extends AbstractProject> extends AbstractProjectAction implements ProminentProjectAction { public AbstractProjectReport(T project) { super(project); } public String getIconFileName() { for (AbstractBuild build = getProject().getLastBuild(); build != null; build = build.getPreviousBuild()) { final AbstractBuildReport action = build.getAction(getBuildActionClass()); if (action != null) { return PluginImpl.ICON_FILE_NAME; } } return null; } public String getDisplayName() { for (AbstractBuild build = getProject().getLastBuild(); build != null; build = build.getPreviousBuild()) { final AbstractBuildReport action = build.getAction(getBuildActionClass()); if (action != null) { return PluginImpl.DISPLAY_NAME; } } return null; } public String getUrlName() { for (AbstractBuild build = getProject().getLastBuild(); build != null; build = build.getPreviousBuild()) { final AbstractBuildReport action = build.getAction(getBuildActionClass()); if (action != null) { return PluginImpl.URL; } } return null; } public String getSearchUrl() { return PluginImpl.URL; } public boolean isGraphActive() { return false; } public Collection getResults() { for (AbstractBuild build = getProject().getLastBuild(); build != null; build = build.getPreviousBuild()) { final AbstractBuildReport action = build.getAction(getBuildActionClass()); if (action != null) { return action.getResults(); } } return Collections.emptySet(); } public Statistic getTotals() { for (AbstractBuild build = getProject().getLastBuild(); build != null; build = build.getPreviousBuild()) { final AbstractBuildReport action = build.getAction(getBuildActionClass()); if (action != null) { return action.getTotals(); } } return null; } protected abstract Class> implements AggregatableAction { public JavaNCSSBuildIndividualReport(Collection results) { super(results); } @Override public synchronized void setBuild(AbstractBuild build) { super.setBuild(build); if (this.getBuild() != null) { for (Statistic r : getResults()) { r.setOwner(this.getBuild()); } } } public MavenAggregatedReport createAggregatedAction(MavenModuleSetBuild build, Map<MavenModule, List> moduleBuilds) { return new JavaNCSSBuildAggregatedReport(build, moduleBuilds); }}That was fairly painless… Note that we interfaces for both the freestyle and maven2 project types, this is OK as the freestyle projects will ignore the Maven2 stuff and vice-versa while the common code is shared by both. Next we need the aggregated build report:package hudson.plugins.javancss;import hudson.maven.*;import hudson.model.Action;import hudson.plugins.javancss.parser.Statistic;import java.util.ArrayList;import java.util.Collection;import java.util.List;import java.util.Map;public class JavaNCSSBuildAggregatedReport extends AbstractBuildReport implements MavenAggregatedReport { public JavaNCSSBuildAggregatedReport(MavenModuleSetBuild build, Map<MavenModule, List> moduleBuilds) { super(new ArrayList()); setBuild(build); } public synchronized void update(Map<MavenModule, List> moduleBuilds, MavenBuild newBuild) { JavaNCSSBuildIndividualReport report = newBuild.getAction(JavaNCSSBuildIndividualReport.class); if (report != null) { Collection u = Statistic.merge(report.getResults(), getResults()); getResults().clear(); getResults().addAll(u); getTotals().add(report.getTotals()); } } public Class> implements ProminentProjectAction { public JavaNCSSProjectIndividualReport(AbstractProject project) { super(project); } protected Class<? extends AbstractBuildReport> getBuildActionClass() { return JavaNCSSBuildIndividualReport.class; }}Don’t repeat ourselves comes in handy here as essentially all the work has been done for us!. The project aggregated report:package hudson.plugins.javancss;import hudson.model.Actionable;import hudson.model.ProminentProjectAction;import hudson.model.AbstractBuild;import hudson.model.Action;import hudson.maven.MavenModuleSet;import hudson.maven.MavenModuleSetBuild;import hudson.plugins.javancss.parser.Statistic;public class JavaNCSSProjectAggregatedReport extends AbstractProjectReport implements ProminentProjectAction { public JavaNCSSProjectAggregatedReport(MavenModuleSet project) { super(project); } protected Class<? extends AbstractBuildReport> getBuildActionClass() { return JavaNCSSBuildAggregatedReport.class; }}Again DRY to the rescue… At this point all that remains is to present the reports from these backing objects… so on with the jelly views. The helper classes and our inheritance makes this easy… all we need is two jelly files: hudson/plugins/javancss/AbstractBuildReport/reportDetail.jelly and hudson/plugins/javancss/AbstractProjectReport/reportDetail.jelly. Here they are:<j:jelly xmlns:j=“jelly:core” xmlns:st=“jelly:stapler” xmlns:d=“jelly:define” xmlns:l="/lib/layout” xmlns:t="/lib/hudson" xmlns:f="/lib/form"> Results Package Classes Functions Javadocs NCSS JLC SLCLC MLCLC Totals ${it.totals.classes} ${it.totals.functions} ${it.totals.javadocs} ${it.totals.ncss} ${it.totals.javadocLines} ${it.totals.singleCommentLines} ${it.totals.multiCommentLines} <j:forEach var=“r” items="${it.results}"> ${r.name} ${r.classes} ${r.functions} ${r.javadocs} ${r.ncss} ${r.javadocLines} ${r.singleCommentLines} ${r.multiCommentLines} </j:forEach> </j:jelly>Yep, the two files are identical! Other plugins may not be quite so lucky… but in general the project level report should be the same as the report for the latest buildMaking a pluginNow we are ready to make our plugin…. for this we need a class that extends hudson.Plugin and registers our publisher’s descriptors with the appropriate lists… here it is:package hudson.plugins.javancss;import hudson.Plugin;import hudson.maven.MavenReporters;import hudson.tasks.BuildStep;public class PluginImpl extends Plugin { public void start() throws Exception { BuildStep.PUBLISHERS.add(JavaNCSSPublisher.DESCRIPTOR); MavenReporters.LIST.add(JavaNCSSMavenPublisher.DESCRIPTOR); } public static String DISPLAY_NAME = “Java NCSS Report”; public static String GRAPH_NAME = “Java NCSS Trend”; public static String URL = “javancss”; public static String ICON_FILE_NAME = “graph.gif”;}And that’s pretty much it… we should have a working pluginFinishing touchesOK, so the plugin does not have health reports (i.e. the weather icons) and it does not show a trend graph… I think I’m going to need a part 8 :-(
