Sourceware 25 Roadmap
Sourceware has been running for almost 25 years, providing a worry-free, developer friendly home for Free Software core toolchain and developer tool communities. And we would like to keep providing that for the next 25 years.
That is why in the last couple of years we have started to diversify our hardware partners, setup new services using containers and isolated VMs, investigated secure supply chain issues, added redundant mirrors, created a non-profit home, collected funds, invested in open communication, open office hours and introduced community oversight by a Sourceware Project Leadership Committee with the help from the Software Freedom Conservancy.
Please participate and let us know what more we (and you!) can do to make Sourceware and all hosted projects a success for the next 25 years.
The first 25 years
Almost 25 years ago, on 6 September 1998, Sourceware came online. Then called sourceware.cygnus.org. The first project being hosted was GCJ (GNU Compiler for the Java programming language). Soon followed by Cygwin and eCos. Many of the early hosted projects were part of the GNU Toolchain. Automake, autoconf, libstdc++, gsl, gdb, binutils, glibc, guile. And in October 1999 they were joined by GCC, which had been running its own server setup for the EGCS project. This is why the Sourceware project is also often referred to as the GNU Toolchain Infrastructure. But there were also lots of non-GNU projects joining in these early days. Like mauve, newlib, libffi, xconq, docbook-tools, insight and bzip2.
At the start Cygnus made clear that it believed the important thing was having a diverse community participate in Sourceware (an alternative term for Free Software or Open Source) infrastructure and projects.
Cygnus believes that software infrastructure should be free ... Participation of organizations and individuals around the world is critical. ... Over time, we will be hosting additional technologies and driving new initiatives. And what we call infrastructure today will continue to grow. Thanks for participating in this community.
After Red Hat took over, they continued this tradition. The server was briefly called sources.redhat.com, from July 2000 to November 2001, after which it was called soureceware.org (a domain not controlled by Red Hat). And today some of the hosted projects have their own domain names, cygwin.org, dwarfstd.org, elfutils.org, gcc.gnu.org and valgrind.org.
Red Hat set up OSCI, Open Source Community Infrastructure. The main Sourceware servers are hosted in what is called the Community Cage, co-location services for various community projects. Red Hat provides the hardware and network connectivity, but the servers themselves are community managed.
After the FSF setup their own forge, savannah, some GNU projects moved there, but most stayed and other core toolchain and developer tool projects joined Sourceware. These days about 20 active projects call Sourceware their home. Projects like Cygwin, a UNIX API for Win32 systems, the GNU Toolchain, including GCC, the GNU Compiler Collection, two C libraries, glibc and newlib, binary tools, binutils and elfutils, debuggers and profilers, GDB, systemtap and valgrind. Sourceware now also hosts standard groups like gnu-gabi and the DWARF Debugging Standard.
Preparing for the next 25 years
A couple of years back we realized that users seemed happy, but the volunteers who drive Sourceware were a bit invisible. The services provided were mostly taken for granted. And while things were stable and Red Hat continued to be a generous sponsor of the hardware and connectivity, it is always good to think about any future needs and changes.
So we set out to be a little bit more open and public. Starting a couple of initiatives we hoped would help sustain Sourceware for the next 25 years.
Being more open and public
When Sourceware was started everybody knew the overseers and the mailinglist where you could ask any infrastructure question. But over the years this knowledge didn't spread. Although not a secret, the mailinglist was hidden. And only old-timers remembered. This was partly because the list was also for administrative requests.We fixed this by creating a really private admin-requests mailinglist and making the overseers mailinglist fully public for community discussions.
We also created a bugzilla sourceware infrastructure component, so requests could be better tracked.
We now publish public technical roadmaps and provide quarterly updates of progress and newsworthy infrastructure updates.
There is also an irc channel #overseers on irc.libera.chat. Overseers Open Office hours take place in the same irc channel every second Friday of the month at UTC 18:00.
And finally announcements, notices about downtime and temporary issues with our network are also posted to the fediverse @sourceware@fosstodon.org
The Sourceware mission page now lists various ways to contact and participate in the community.
Diversify hardware and service partners
Although Red Hat has been a reliable and trusted hardware sponsor, it is always a good idea to make sure there are options and alternatives.
So we now have a collection of corporations and organizations (Red Hat, OSUOSL, Brno University of Technology, Marist College, IBM, Works on Arm initiative, Gentoo Foundation) providing hardware.
We also are working more closely with OSUOSL and the FSF tech-team who both have paid staff to provide services. OSUOSL provides various buildbot machines and sometimes picks up services that we have less experience with, like a mattermost server. The FSF tech-team already provides various services for GNU projects and has been helping with email services and providing help setting up a secure signed release process.
Expanding the services
Most Sourceware projects use email based git workflows. As explained in the Sourceware - GNU Toolchain Infrastructure roadmap, we are trying to make such workflows more fun, secure and productive by adding new services.
For git we introduced cgit, a hyperfast web frontend for git repositories, for sourceware, cygwin, dwarfstd and gcc. And it is now possible for hosted projects to use gitolite to use fine-grained access control to their repositories. Both cygwin and dwarfstd already do use gitolite.
To improve email handling we installed a public-inbox instance at inbox.sourceware.org. This allows people to git clone all mailinglist and read them via NNTP, IMAP, Atom feeds or HTML archives
We have a shared buildbot for Sourceware projects at https://builder.sourceware.org/. This includes compute resources (buildbot-workers) for various architectures thanks to some generous sponsors. We have native/VM workers for x86_64, ppc64le, s390x, ppc64, i386, arm64 and armhf for debian, fedora and centos and x86_64 container builders for fedora, debian and opensuse. The buildbot has try builders - "pre-commit", CI builders - "quick regression tests", and full builders - "all (slow) tests". All test result go into bunsen.
For patch tracking there is patchwork.sourceware.org.
Thanks to OSUOSL we now have snapshots.sourceware.org to publish static artifacts from current git repos created in isolated containers. It can be used as alternative to git hooks or cron jobs to generate snapshots for things like:
- poke code and doc snapshots
- elfutils code coverage
- libabigail website, manuals and api docs
- valgrind snapshots and manuals
- DWARF standard draft spec
The container files and build steps are defined through the builder project.
We also added more ftp, rsync and http mirrors. And added sourcehut mirrors at https://sr.ht/~sourceware/ for all git repos. This allows anybody to fork any sourceware project on sourcehut, prepare patches and submit a merge request through email without having to locally setup git send-email or smtp - the patch emails are generated server side. Finally all active (and inactive) projects are now archived at the Software Heritage (including old cvs and subversion repos).
There is full list of Free Infrastructure Services provided by Sourceware. Also broken down for each hosted project.
Reflections on Trusting Trust
Years ago Ken Thompson laid out the roadmap for attacking an operating system via the compiler and other code generation tools. These days these are known as supply chain attacks. The Free Software community should reasonably insist that they be defended against these kinds of attacks with mechanisms for prevention, detection and restoration.
Sourceware now offers different ways to attest a patch or email is valid. Using the Sourceware public-inbox instance you can use b4 for patch attestation using dkim, gpg-signed emails or patatt.
Projects concerned with source code integrity now have various options to use signed git commits, signed git pushes, or use gitsigur for protecting git repo integrity.
A non-profit home
Sourceware is a home to various project communities. But it didn't have a (fiscal) home itself. This made it difficult to grow as an independent organization. Individual contributors might need to take on tasks and responsibilities that would be better handled by a real non-profit organization.
After almost 10 months of discussions Sourceware officially joined the Software Freedom Conservancy as member project this year.
Now Sourceware can receive earmarked donations. Conservancy can hold any assets (domain names, physical computer equipment, etc.) in the name of Sourceware, negotiate and execute contracts, and provide other fiscal sponsorship services. While community members avoid any non-profit administrivia.
Sourceware will leverage Conservancy advisory role in how community projects are impacted by and can comply with recent regulations like the USA Cyber Security Directives and the EU Cyber Resilience Act.
And with Conservancy as fiscal sponsor, Sourceware will also be able to have the community of volunteers work together with paid contractors and enter into contracts for managed infrastructure where appropriate.
Funds
One of the benefits of having a fiscal home at the Software Freedom Conservancy is that the community can now receive funds and hold its own assets. Even though we are only officially a member project for a couple of months and we didn't do any real fundraising campaign yet, we already got enough donations for an emergency hardware replacement fund. In case one of our hardware partners would suddenly and unexpectedly drop support we can now simply replace any hardware.
Donations can come from individuals, corporations or through grants (tax-deductible in the USA). To help Sourceware you can become a Software Freedom Conservancy Sustainer or contribute money directly by mentioning "Sourceware" on the comment or memo line when donating.
Community oversight
With great community expectations comes great responsibility. On advise of the Software Freedom Conservancy we created a Sourceware Project Leadership Committee (PLC). Each individual member of the PLC participates in their own capacity, but nevertheless the majority of the PLC never includes a majority of people affiliated with the same organization to prevent corporate politics. The Conservancy also has a strict Conflict of Interest Policy.
The PLC members include various volunteers who have helped run Sourceware and the various hosted projects. The current PLC is eight members strong:
- Frank Ch. Eigler
- Christopher Faylor
- Ian Kelling
- Ian Lance Taylor
- Tom Tromey
- Jon Turney
- Mark J. Wielaard
- Elena Zannoni
They meet once a month to discuss any community input, set priorities and decide how to spend any funds, negotiate with hardware and service partners, create budgets together with the Conservancy, or decide when a new fundraising campaign is needed.
The next 25 years
We hope the above initiatives created a solid foundation for Sourceware and all the hosted core toolchain and developer tool project communities.
Please participate and let us know what more we (and you!) can do to make Sourceware and all hosted projects a success for the next 25 years.