Hudson's Bright Future


February 3, 2011 By Jason van Zyl

We believe that Hudson users can look forward to a long, bright future.

Working with the community, Oracle and Sonatype are each putting a number of full-time engineering resources on Hudson. The Hudson lead, Winston Prakash from Oracle, is highly skilled, very thoughtful, and he cares about the community. He is also the first person to create detailed, comprehensive architectural documentation.

This kind of documentation (which has never been available in the past) is required to understand how Hudson can be improved. The lack of architectural documentation, along with how decisions were made, left the Hudson community mostly dependent on a single individual for core changes.   Let’s be honest about where this led:

- The Hudson WAR distribution incorporates over 100 dependencies, a troubling number of which are forked.   As result, there is no easy way to incorporate improvements from those forked projects in Hudson. It’s also not very easy to know why they were forked in the first place.

- The core technology for the user interface in Hudson is Jelly. We used Jelly in Maven 1.0, but took it out more than five years ago for Maven 2.x because it is not maintained and very difficult to work with. Hudson is still dependent on Jelly, but there are many more standard, and flexible, options for the UI technology: choices like GWT, Vaadin, or JSF.

- Hudson has had minimal project infrastructure. There are extremely limited unit tests, and integration tests that leave a lot to be desired (e.g. they don’t run on recent versions of the Mac OS X). Weekly builds of Hudson without a sophisticated automated test infrastructure is just not wise. Winston and I will propose a plan to clean up some of the testing infrastructure as a first priority. We’re not in a rush to push out releases where the community has to act as QA. It’s great to get feedback from users but we think a better job can be done on the testing front to alleviate this burden on users. It’s not glamorous, it takes a lot of time, but it’s necessary if you value stability and quality.

- Little to no effort has been made to track the provenance of the code, and there are numerous licenses that are used throughout the codebase. The licensing issues can likely be resolved using different libraries, and doing some modularization, but the provenance and IP issues are of utmost importance when you care about downstream consumers. These issues are of little or no concern for SaaS providers, but critical for companies that need to deliver software used on-premise.

So, I’ve told our developers to stay focused on improving Hudson’s infrastructure and fundamentals. Here’s our current work log:

- Testing infrastructure, and bolster the unit and integration tests
- JSR330 support for plugins
- JAXRS support for web services
- Maven 3.x support (94% of Hudson installs also use the Maven plugin)
- Eclipse integration
- Netbeans integration
- Assessment of the current Hudson architecture
- JSR330 architecture work to modularize the Hudson core
- Using a standard view technology like GWT, Vaadin, or JSF
- Inspection of non-standard/forked dependencies and trying to re-align with current mainstream releases
- Isolating code with problematic licenses and trying to reduce the license footprint
- Using standard Maven plugins for creating ancillary Hudson distributions

I’ll describe all of this in greater detail in subsequent blog posts.

  • Tetsuo

    What about Jenkins?

  • http://twitter.com/jlorenzen James Lorenzen

    Very nice. Didn’t realize it was that bad. Ripping out jelly sounds like a very nice improvement. Never really understood it in the first place. Don’t existing plugins that change the hudson UI “extend” jelly. What happens to those existing plugins once the hudson UI is changed?

    • http://www.facebook.com/brianefox Brian Fox

      There are some options for adapting existing plugins to a more modern framework programmatically. You’ll hear more on that soon.

  • Gaby

    Good luck guys!
    So many work with M2Eclipse, Nexus, Tycho and so on!

  • Pelegri

    I’ll comment on the topic I know first hand: provenance of the code – you don’t indicate whether you mean core or plugins, so I’ll assume you mean the core, as that’s what’s most relevant.

    The answer I gave you before [1] applies: Kohsuke and I had started a cleanup pass at Sun and captured most SCA/OCA. I completed that pass in mid 2010 to help Winston/Ted after Kohsuke left; that included SCA/OCAs, and I did a ToI to Winston and Susan when I left Oracle.

    [1] http://hudson.markmail.org/search/?q=pelegri#query:pelegri%20from%3A%22Eduardo%20Pelegri-Llopart%22%20order%3Adate-backward+page:1+mid:ofcaal3wmfjkaqgh+state:results

  • http://openid.javageek.org/javageek JavaGeek

    I don’t understand. If Hudson is really such a poorly designed/implemented project, why not start it from scratch then? Take all the feature list from Hudson, and do a clean room implementation that also takes into consideration your improvement requests.

    • jvanzyl

      We have many customers that use Hudson. The plugin API will need to be supported forever. Starting from scratch is not an option. Our strategy is very much akin to what we did for the Maven 2.x to Maven 3.x transition. Create a pathway for new innovation while supporting legacy APIs in a binary compatible way. We’ve demonstrated it can be done. For the adherence and testing of backward compatibility we will take our cue from the Eclipse Platform.

  • Pelegri

    JSR 300? DRM for JavaME? :)
    re: Jelly and Web Technologies, I’ve been in so many discussions about what is a good and what is a bad web technology that I not even touch them. But it would be interesting to see Oracle endorse Google’s GWT…

    • jvanzyl

      They already did as an option. That’s not to say we are going to use it as that will be a discussion but it’s ASL so it’s fine. GWT is what we used for our Maven 3.x integration and that’s working in a hybrid Jelly/GWT mode. We’ll need something Jelly compatible for a long-time but we at Sonatype have a preference for working with GWT.

      • Pelegri

        Well, Google’s GWT is kind of AWT, uses some of the AWT IP/APIs but it really is not AWT. Which sounds a lot like another situation over which Google and Oracle have disagreements.

    • http://www.geoffreywiseman.ca Geoffrey Wiseman

      Yes, the JSR300 reference is confusing. I’m not even sure which JSR was intended to be referenced, but DRM for JavaME seems unlikely. ;)

      • http://www.discursive.com Tim O’Brien

        We meant JSR-330 for the core modularization effort.

    • jvanzyl

      JSR330, not JSR300 :-)

  • Kohsuke Kawaguchi

    I guess I’m that single individual you refer to.

    I think saying that core changes are “mostly dependent on a single individual” isn’t a fair statement to all the others that have contributed [1], but I certainly accept the blame for the less-than-optimal architecture choices. Yes, there are some regrets I have for the choices that are made, which I think is true for any project of this size and history.

    With respect to dependencies, I think calling them “fork” is an over-statement. There are about 8 or 10 libraries that are critical to the functionality that we needed to patch them. They closely track the upstream libraries (where there are active upstream development), bugs are reported to the upstream, etc. It is often the case that there are good but not perfect libraries — I find it more practical to use them with some modifications, rather than reimplement from scratch. It’s simply faster than my writing the same thing all over.

    With respect to Jelly, my feeling is more mixed. Many design decisions in Hudson were driven by the extreme degree of pluggability that the plugin system called for, and that takes something like Jelly, whose behavior can be modified at very deep level. And the good news is that that same flexibility allows us to separate the XML syntax frontend and the XML generation backend. so I think some pluggability here to enable different “frontend” would work well. I’d like to push this further indeed.

    But beyond those individual technical issues, I’d like to point out that the software design is a balancing act of many factors. And overall I think I’ve made good enough calls, or else we aren’t seeing 300+ plugins developed today. And because we now carry such a large ecosystem, I’ve thus far resisted the temptation to throw away everything and do it from scratch again. I believe more in incremental, non-disruptive changes, made little by little over time (although it could be that you are saying the same thing.)

    [1] https://github.com/jenkinsci/jenkins/graphs/impact (scroll to the right for the current changes)

    • Peterfipps

      Well done for writing it. Any architecturally sound projects that I have ever seen, nobody ever uses. Its all about the balancing act, between shortcuts and maintainability. It hard to get this right, and clearly you have.

    • jvanzyl

      The architectural issues would be there regardless, and we would have addressed those issues regardless. This is but one of many posts about why we chose the path we did. I can understand why you think the way you do, and why you did what you did philosophically — which is the real crux about what I disagree with you about. Technically I agree with you, projects grow organically and it’s a balancing act. I’m certainly not advocating starting from scratch, that would be impossible because of the user base.

      • optilude

        So why not work with the Jenkins project to make the relevant improvements?

    • Kohsuke Kawaguchi

      One more note, as others posted, I’m sad that Sonatype chose not to engage the community to work on these technical agendas. We’ve seen multiple announcements as far back as 2 years ago [1,2] that you guys are interested in some changes in Hudson, but there never was a single line of code drop, and we were not even given a chance to discuss those.

      You’ve cited some kind of IP / CLA (contributor license agreement) issues [3] for holding things back, despite CLA and signatories visible [4] in pubic. Yet now that the whole Hudson team moved on to the Jenkins project, everything is suddenly good and you can start contributing the project. Hmm. (BTW, I haven’t seen your name on the CLA signatories — is the up-to-date list maintained somewhere else now?)

      I’m still hoping that you would reconsider your decisions and work with the Jenkins community. We’d love to have the actual technical discussions, and I think we’ve been good at resolving those in the 7 years of the project history. That, to me, is in the best interest of the user community.

      [1] http://jenkins.361315.n4.nabble.com/Plexus-introduction-td391595.html#a391599
      [2] http://jenkins.361315.n4.nabble.com/Sonatype-s-Hudson-Plans-for-Maven-Integration-td391581.html#a391589
      [3] http://jenkins.361315.n4.nabble.com/Re-Adding-me-as-co-owner-of-Hudson-java-net-project-td3023075.html#a3024526
      [4] https://sca.dev.java.net/CA_signatories.htm

  • http://twitter.com/0x4C4A0A46 0x4C4A0A46

    @jvanzyl Given your past statements – http://bit.ly/gtenBg – I’m quite curious when the IP got resolved, elucidate please? I guess I fail to see how the source IP situation is any different than it was whenever you decided to withhold contributions from the original Hudson project. What’s changed?

    You’ve flip flopped without any outward change other than core Hudson developers leaving the project. Perhaps no F/OSS project can have more than one BDFL attached? I’m not sure, but for people that are not just drive-by observers of this entire situation, but have been attached for a long while, your behavior has a very bad smell….

    • jvanzyl

      My position has not changed at all, but I’m not sure why you conflate that with plan to resolve the IP issues. I would consider what the Eclipse Foundation does as the gold standard and we’re a long way from that. To do that will not be a small amount of work and I believe the only viable paths are to move Hudson to the Eclipse Foundation or work with Oracle. Brian and I talked with talked with Kohsuke about moving the project to the Eclipse Foundation. Oracle was not interested in moving it to Eclipse at that time, and I understand that position. I also talked to Kohsuke about forking it to Eclipse and Kohsuke said to me, verbatim over IM, that “I think Oracle is a good place for Hudson.”

      I don’t know what transpired between then and now exactly, but I can’t help but think Cloudbees influenced the decision to move away from Oracle and I personally think it is a bad one. Anyway, just after suggesting the move to Eclipse I met Ted. He gave me his perspective, told me that trademark usage would be liberal (and it is), and explained that an IP clean core was a requirement of theirs. I have not found Ted to be unreasonable at all.

      So a plan to sequester parts of Hudson and move toward creating a pristine core is one of the things Winston and I will be working on. If you take a look at the minutes for Jenkins governance meeting (http://meetings.jenkins-ci.org/jenkins/2011/jenkins.2011-02-04-23.02.log.html) as ardent and good intentioned as they are, it’s hard for me to believe that without the help of a large organization, like Eclipse, IBM or Oracle, that anything useful will be accomplished in this regard. I also doubt at this point whether many groups want to risk entaglement with Oracle.

      As for witholding code, people will always speculate but of our JSR330 work JAXRS only the JSR330 work is in a form where we can actually contribute it back. Additionally all the code is going to be MIT licensed so I’m not sure how it could ever be witheld from Jenkins if that’s what you’re referring to. If on the Hudson side folks wanted to setup a cross-project CLA system I would be all for that. Even at this point I don’t think there have been any irreversible decisions and I would encourage anything like a cross-project CLA plan that allows the projects to merge again under the Hudson banner some day.

      • Pelegri

        > plan to resolve the IP issues.

        What “IP Issues”, Jason? See my earlier comment.

        > but I can’t help but think Cloudbees influenced the decision to move
        > away from Oracle

        Nope.

        > it’s hard for me to believe that without the help of a large organization, like
        > Eclipse, IBM or Oracle, that anything useful will be accomplished in this
        > regard.

        > I also doubt at this point whether many groups want to risk
        > entaglement with Oracle.

        Those are interesting opinions…

      • Sacha Labourey

        Jason,

        I think you and Oracle are giving way too much credit to CloudBees in what happened, but this is giving us great visibility, so thank’s.

        As for the IP matters, ORCL could have changed things for a long time – I don’t see this as related to Kohsuke – so spinning this is as a “Jenkins == dirty IP, Hudson == clean IP” is far from the truth, especially when I see some of the recent code git-pulled from the Jenkins codebase into the legacy-Hudson codebase. Setting up “clean IP” processes, doesn’t require a large corporation. Those are actually pretty simple processes. It is just that projects must enforce them and people have to respect them. Lastly, when it comes to “IP”, you have tastes and colors: ORCL’s lawyers don’t like the EPL nor the LGPL. Fine. But I don’t think this opinion is necessarily shared by everybody.

        As for the trademark, Ted never agreed to give a life-long policy on how this TM could be used. He might have offered a liberal license to Sonatype, great, but that wouldn’t cover the community at large. And so the terms under which he could grant that brand – that you consider “liberal” today – could have changed into much more restrictive terms anytime. And while I am sure Ted would stick to his words, people come and go. So this is not about Ted’s words, it is about a legally binding agreement with ORCL.

  • Alex Blewitt

    I’m not sure how JSR300, the DRM API for J2ME helps modularize the core?

    Have you considered migrating the plugin system to a more mature pluggable runtime system like OSGi?

    Finally, now that Hudson and Jenkins are moving in different directions, will we see the gratuitous ux bug fixed whereby successful runs are represented as green rather than blue, to be consistent with every other success/fail tool? I gather this has been resisted for some time by those who are no longer involved with Hudson.

    • http://lacostej.blogspot.com/ Jerome Lacoste

      there’s a plugin for changing the color.

    • Al

      The “gratuitous ux bug” is pretty helpful to users who can’t easily distinguish red from green. Of course the real bug is that success and failure should not be distinguished by colour alone.

  • Christian Schneider

    With all the unrest in regard to your changes in maven central it would have been a good decision for Sonatype to support the community and rather drive Jenkins forward than Hudson.

    • brianefox

      While we have disagreements about the architecture and future direction of Hudson, this decision is orthogonal to Maven Central. We have been in discussions with the Jenkins team to ensure continued access to Central for the benefit of all users. We have offered (as we do to any qualified OSS project) a free license to Nexus Pro and will work with them to get the required quality checks in place so that they can get their artifacts into Maven Central on the same terms as Apache, Codehaus, Exo Platform, Oracle and many other OSS Forges.

      Sonatype has provided resources behind Maven Central that improve the situation for all users, not just those building with Maven. In recent years, many improvements have been provided to the community:

      *new redundant mirrors have been added in Europe (http://www.sonatype.com/people/2010/10/new-official-maven-central-repository-in-europe/)

      *a new staging machine has been setup to isolate and scan inbound artifacts before they are published

      *free repository hosting which has enabled close to 1000 projects to easily publish to central (http://nexus.sonatype.org/oss-repository-hosting.html)

      *automated bundle processing that cut the time to load 3rd party artifacts down from 6 weeks to 1 day (https://docs.sonatype.org/display/Repository/Uploading+3rd-party+Artifacts+to+Maven+Central)

      *a new real-time search service (http://mavencentral.sonatype.com/)

      In the next few weeks, the Maven Central architecture will undergo yet another improvement as we move to a Private cloud based solution that isolates these critical services from hardware failure and allows it to scale horizontally

      We will have to agree to disagree on certain decisions at times, but Sonatype will continue to support the overall community via our contributions to Apache Maven, m2e, Tycho, Nexus, Async HTTP, Guice, P2, and Maven Central

  • http://greybeardedgeek.net GreyBeardedGeek

    Sounds good,
    But please, don’t do JSF.
    GWT, fine, Vaadin, great – even plain ‘ol JSP, but please don’t saddle the project with JSF.

    Thanks.

    • jvanzyl

      I actually like Javascript, but I couldn’t convince anyone else :-) I think for this type of project, it’s Java developers making contributions in Java so if it’s something more Java-centric like GWT it will be a win I think. For me, my favourite combination right now is Sitebricks using ExtJS as the front-end. I’m still a little taken aback at how many Java developers really dislike Javascript.

  • Phil

    While cleaning up licensing and code provenance issues makes sense for companies that want to make money, let’s not pretend that this is some altruistic effort to help the community.

    “the provenance and IP issues are of utmost importance when you care about downstream consumers. These issues are of little or no concern for SaaS providers, but critical for companies that need to deliver software used on-premise.”

    Provenance and IP issues are also important when you care about charging enterprise contracts to companies that are required to have IP indemnification. I have no problem with this, btw–I like it when developers get paid for their efforts.

    But as you said–let’s be honest about it.

    • http://www.discursive.com Tim O’Brien

      This is ridiculous, i guess when projects go through the process of making sure that all IP issues are solved at the Apache Incubator that you would raise the same objection. “Oh, you are only doing this because you want to sell a product to enterprise customers”. What you fail to realize is that when a company like RedHat, Oracle, or IBM spends time, money, and effort making sure that the IP is bullet proof that this effort ends up benefiting the community as a whole. You wouldn’t be using Linux today were it not for the fact that vendors like RedHat took the time to make dead certain that all of the t’s were crossed and the i’s dotted. Was the code contributed by people with proper CLAs, is there a CCLA on file? I guess you only think that’s important if you want to sell software?Like trademarks, copyrights, and licenses, all of these “legal” things that developers don’t understand are the “vehicles” that allow us to even have a healthy open source ecosystem.

      • Phil

        Tim, Thanks for the reply.I’m pretty sure I wouldn’t be using Linux today if it were not for Linus Torvalds; Red Hat, I’m not so sure about. ;)I’m willing to be educated about the need for proper IP review and code provenance. Can you explain to me how a missing CLA would prevent me from compiling and running the code, or modifying the source and contributing my changes back to the community?

        • http://www.discursive.com Tim O’Brien

          Undoubtedly Linus made important contributions, but I think Alan Cox’ contribution to the Linux Kernel played a really important role. I don’t think you’d be running Linux if it were for Alan Cox, and he’s a core developer from RedHat. The success of the Linux kernel is a perfect example, that effort succeeded because a collection of interested players could participate. Corporations, individuals, whoever wanted to play could play. It wasn’t because of the mailing lists or the source control it was because of the license. The core feature of open source is that we’ve come up with legal structures that encourage sharing while protecting contributions.

          A CLA isn’t for the benefit of the developer, it is for the benefit of every other developer and user. It is a statement that you, as a developer, are making contributions to the project and that these contributions are not someone else’s property. Apache takes this even one step further and requires most employers to sign a CCLA. It isn’t just a convenience for corporations.

          • Spacemoose

            > A CLA isn’t for the benefit of the developer, it is for the benefit of every other developer and user.

            Not really; CLA is for benefit of distributor of the code, typically a company. Not that there is anything wrong with that. But CLA is really for licensing (and hence, distribution) rights, not for right to use which would affect users.

          • http://www.discursive.com Tim O’Brien

            You know what Spacemoose, you are totally right. The CLA is only there for distributors. It isn’t there to make sure that other developers don’t waste a few years contributing to a project that ultimately gets dragged into a lengthy code review to determine what must be rewritten or discarded due to murky questions about who worked for whom when.

  • http://draconianoverlord.com shaberman

    I like GWT as much as the next Java developer, but my naive understanding of Hudson’s use of Jelly was that plugins could arbitrarily contribute HTML/functionality to the view layer. AFAIK, this is not something that’s really compatible with GWT’s “all source code is known at compile-time” approach.

    So I’m somewhat skeptical it could provide the same role that Jelly (or a similar dynamic language) did. Though I’d love to be proven wrong. Have you guys have seen or spiked ways of GWT providing an extensible view API that plugins can semi-arbitrarily, at runtime contribute to?

    • brianefox

      Maintaining compatibility with existing plugins as well as ease of plugin development are key considerations that need to be resolved before a different framework for the core can be decided. This is something we’ve been looking at for a long time at Sonatype as we built out our commercial version, and this discussion will be done on the dev list for everyone to participate in.

  • http://www.facebook.com/jimbethancourt Jim Bethancourt

    Although I don’t use Hudson or Jenkins, the idea of having JSR 330 based plugins and a common plugin API / SPI for plugin implementers to implement against makes a lot of sense so they can be more easily be used on either platform. It would make a lot of sense to have a situation where the underlying platform can be switched out without impacting the success of the build. Perhaps this is where the two teams could come together and agree on a set of APIs / SPIs that the plugin developers would develop against, and let the underlying platforms compete on price, performance, and features, while the users end up as the true winners without losing an entire plugin ecosystem.

  • V Volle

    I understand the criticsm w.r.t. Hudson, but I do not understand why you chose Hudson over Jenkins. All the problems could be solved in Jenkins as well as in Hudson. So what does it really come down to? More influence (that is not meant as a critiique, That is a very good reason) but if that is the main reason, you should say so more clearly. What else?

  • http://hwellmann.blogspot.com/ Harald Wellmann

    I can see a bright future for Jenkins, now that there’s a clear line between the community and Oracle.

    And I can see a maybe not so bright future for Sonatype, after taking sides with Oracle, the company that has succeeded in alienating an unprecedented number of open source communities within less than a year.

    There is a technical and a political aspect to the Hudson/Jenkins story.

    While I agree to some of the technical issues you raise – lack of documentation and lack of stable releases in particular – I have no reason to believe that Sonatype will now be the Saviour to heal all sores of Hudson. Some projects sponsored by Sonatype (m2eclipse, Tycho) are in no better shape in this respect.

    Besides, you might just as well have chosen to support the Jenkins community to join in building the best CI server available.

    Hudson is an open source project, and even more so, a community project, with a significant part or even the majority of the codebase coming from the community at large and not from a single corporate environment (completely unlike Glassfish, for example).

    An overwhelming majority of the former Hudson and now Jenkins community has voted for independence and against a “benevolent dictatorship” (and some people would drop the “benevolent”, I believe).

    There used to be a company with the tagline “The Maven Company” – just in case anyone forgot: Maven is an open source project under the umbrella of the Apache Software Foundation, a body that recently left the so-called Java “Community” Process, for lack of community in the process.

    Many open source users or developers have turned away from Oracle – did you ever consider the same might now happen to Sonatype?

  • http://nfisher.myopenid.com/ Nathan

    Might I refer everyone to Nat Pryce’s article on “Complaining about Other People’s Code”. Great to hear about the progress being made to understand the system!

    http://www.natpryce.com/articles/000756.html

  • Bruce Phillips

    Jason – I have alot of respect for Sonatype and what you’ve done supporting Maven. If it wasn’t for the excellent Maven books your company provides at no cost and your work on m2eclipse, my team of developers would still be using Ant.

    What I don’t understand after reading your post and the comments is why Sonatype choose to partner with Oracle on Hudson instead of providing development support to Jenkins? I’ve read all the Hudson posts on this blog but didn’t find a clear cut explanation.

  • http://www.facebook.com/warpforge David Timothy Strauss

    “94% of Hudson installs also use the Maven plugin”

    That’s not a useful number because the Maven plugin is installed by default. It’s more meaningful to say, “6% of installations have gone out of their way to disable the Maven plugin.” That doesn’t say much.