Rise and fall of libclamav

Because I was bored and needed to procrastinate, I decided to look at the history of packages using libclamav over the last several releases. This is binary reverse-depends in main on i386:

Release	libclamav soname	rdepends
Sarge	libclamav1	2
Etch	libclamav2	9
Lenny	libclamav5	5
Squeeze	libclamav6	3
Wheezy	libclamav7	3
Jessie	libclamav7	4
Stretch	libclamav7	4
Buster	libclamav7	4

I started working on clamav around Etch (in Ubuntu, so it’s not an exact match) and transitions were a blast back then. Every single soname bump needed significant sourceful changes. It killed quite a number of projects. Of the four we still have in Debian (dansguardian, havp, icap, and python-clamav) only the icap modules aren’t essentially dead upstream.

I guess API stability counts for something if you want people to use your library.

P.S. None of the people working on clamav today are the same as when we had 3 soname bumps in one release cycle.

 

DKIM (DomainKeys Identified Mail) Modernization

When DKIM was originally specified in 2007, 512 bit rsa-sha1 seemed like a great idea (well not a great idea, but not everyone could do rsa-sha256 yet, so rsa-sha1 was let live in the specification to ease transition from Yahoo!’s DomainKeys.  Fast forward to 2012 and suddenly 512 bits wasn’t such a great idea.  The most recent DKIM update the year before had not changed the original 2007 recommendations, but the operational community reacted and 768 bits became a de-facto minimum key length and 1024 bits or more preferred.  All DKIM related packages in Debian were updated to match these more secure requirements.

Roughly a year ago, the IETF (Internet Engineering Task Force) commissioned a new working group, DCRUP (DKIM Crupto UPdate) to look at updating DKIM “to handle more modern cryptographic algorithms and key sizes”.  This has evolved into a combination of throwing out the old and bringing in the new in two separate documents.

The throwing out the old part of the work has been published as RFC 8301.  It removes rsa-sha1 from DKIM and raises the minimum RSA key size to 1024 bits and recommends 2048 bits (before commenting about how horrible 1024 bits is, please read the RFC, it’s there for a reason).

Bringing in the new is still a work in progress, but nearly finished.  It’s being developed in draft-ietf-dcrup-dkim-crypto.  It’s been through working group last call once, and I don’t expect much more change before it’s ready for wider IETF review.  The new is a new signature algorithm, ed25519-sha256.  The ed25519 algorithm is defined in RFC 8032.  It seems to be getting traction in a number of applications.

There are two implementations, that I know of.  For exim, DKIM with ed25519-sha256 support has been committed to their VCS and will (I assume) be in the next feature release.  The other is based on the Python DKIM module dkimpy.  I’ve also written a milter that uses it with Postfix and Sendmail.  We’ve tested against each other and the two implementations are interoperable in our testing.

This is good news for the robustness of ed25519 since Exim is using gnu-tls (as I understand it) and I used libsodium as wrapped by PyNaCl.  Being able to test two dissimilar implementations before the specification is carved in stone has been a big help.  We discovered a few areas that were underspecified and some interesting differences in the different ed25519 variants defined in RFC 8032 (strangely both implementers used the same variant that was not the one in the draft at the time, it’s all fixed now).  I doubt it will surprise anyone that this was two FOSS implementations, proprietary implementers are (other than complaining about having to get off sha1) not heard from.

If you use postfix or sendmail on Debian Stable (Stretch)/Testing/Unstable, you can install dkimpy-milter and try it out.  For Stable, you’ll have to use backports for both dkimpy-milter and some dependencies.

In order to make it easy to try out, I used configuration names and definitions from OpenDKIM (since I think that’s what most sendmail/postfix people use).  If you only used configuration options that are supported by dkimpy-milter, you can just copy over your conf file and start right away (if you used something unsupported, you’ll get an error when you start the service).  In order to use ed25519 signatures, you will need to create an ed25519 key and publish it at a new selector.  You can use the dknewkey script provided by python-dkim (which will be pulled in as a dependency) to generate the key.

As far as I and a few others who have tested this can tell, dkimpy-milter works for the scope of features I’ve taken on so far.  If you try it and have problems, please file bugs.  Similarly, if you options from OpenDKIM that aren’t implemented, please file wishlist bugs so I know what is useful for other people.

So far, I’ve done what was most immediately useful for me.  For my usage, I’ve replaced OpenDKIM and am running it in production on Stretch without issue.  One caution for sysv init users: I don’t use it, so even though I’ve provided a sysv init script, it’s only very, very lightly tested.  If you use sysv init and have problems, please file bugs and I’ll try and fix it (patches would be great).

One final note: please don’t complain about it being on Launchpad and not GitHub.  Unlike GitHub, Launchpad is 100% free software and even though I have issues with the policies for contribution, I still think that counts.

Debian LTS Work March/April 2016

The end of April marks the one year anniversary of my Debian LTS contributions.  Unfortunately it also marks the end (for now at least) of my contributions under the Freexian umbrella.  While I think this is important work, I have always struggled to find time to do it between family, my regular Debian and other free software work, and my other consulting work.  I’ve decided to quit pretending I’ve got the time to provide the consistent contribution that Freexian needs for Debian LTS, so this is my final report.

This report covers two months because I was not well at the end of March (nothing serious, but the timing was very poor).  During this time I worked on preparing to start supporting Wheezy LTS and prepared and tested updating clamav from 0.99 to 0.99.1.  Unfortunately, uploading it to Wheezy is blocked on it getting in to Jessie.  In the mean time, upstream released 0.99.2.  Once that is in Jessie, I plan to upload it to Wheezy as well so this effort will not go to waste, it just has to wait a bit.

Computer System Security Policy Debate (Follow-up)

As a follow-up to my recent post on the debate in the US over new encryption restrictions, I thought a short addition might be relevant.  This continues.

There was a recent Congressional hearing on the topic that featured mostly what you would expect.  Police always want access to any possible source of evidence and the tech industry tries to explain that the risks associated with mandates to do so are excessive with grandstanding legislators sprinkled throughout.   What I found interesting (and I use that word with some trepidation as it is still a multi-hour video of a Congressional hearing) is that there was rather less grandstanding and and less absolutism from some parties than I was expecting.

There is overwhelming consensus that these requirements [for exceptional access] are incompatible with good security engineering practice

Dr. Matthew Blaze

The challenge is that political people see everything as a political/policy issue, but this isn’t that kind of issue.  I get particularly frustrated when I read ignorant ramblings like this that dismiss the overwhelming consensus of the people that actually understand what needs to be done as emotional, hysterical obstructionism.  Contrary to what seems to be that author’s point, constructive dialogue and understanding values does nothing to change the technical risks of mandating exceptional access.  Of course the opponents of Feinstein-Burr decry it as technologically illiterate, it is technologically illiterate.

This doesn’t quite rise to the level of that time the Indiana state legislature considered legislating a new value (or in fact multiple values) for the mathematical constant Pi, but it is in the same legislative domain.

Future of secure systems in the US

As a rule, I avoid writing publicly on political topics, but I’m making an exception.

In case you haven’t been following it, the senior Republican and the senior Democrat on the Senate Intelligence Committee recently announced a legislative proposal misleadingly called the Compliance with Court Orders Act of 2016.  The full text of the draft can be found here.  It would effectively ban devices and software in the United States that the manufacturer cannot retrieve data from.  Here is a good analysis of the breadth of the proposal and a good analysis of the bill itself.

While complying with court orders might sound great in theory, in practice this means these devices and software will be insecure by design.  While that’s probably reasonably obvious to most normal readers here, don’t just take my word for it, take Bruce Schneier‘s.

In my opinion, policy makers (and it’s not just in the United States) are suffering from a perception gap about security and how technically hard it is to get right.  It seems to me that they are convinced that technologists could just do security “right” while still allowing some level of extraordinary access for law enforcement if they only wanted to.  We’ve tried this before and the story never seems to end well.  This isn’t a complaint from wide eyed radicals that such extraordinary access is morally wrong or inappropriate.  It’s hard core technologists saying it can’t be done.

I don’t know how to get the message across.  Here’s President Obama, in my opinion, completely missing the point when he equates a desire for security with “fetishizing our phones above every other value.”  Here are some very smart people trying very hard to be reasonable about some mythical middle ground.  As Riana Pfefferkorn’s analysis that I linked in the first paragraph discusses, this middle ground doesn’t exist and all the arm waving in the world by policy makers won’t create it.

Coincidentally, this same week, the White House announced a new “Commission on Enhancing National Cybersecurity“.  Cybersecurity is certainly something we could use more of, unfortunately Congress seems to be heading off in the opposite direction and no one from the executive branch has spoken out against it.

Security and privacy are important to many people.  Given the personal and financial importance of data stored in computers (traditional or mobile), users don’t want criminals to get a hold of it.  Companies know this, which is why both Apple IOS and Google Android both encrypt their local file systems by default now.  If a bill anything like what’s been proposed becomes law, users that care about security are going to go elsewhere.  That may end up being non-US companies’ products or US companies may shift operations to localities more friendly to secure design.  Either way, the US tech sector loses.  A more accurate title would have been Technology Jobs Off-Shoring Act of 2016.

EDIT: Fixed a typo.

 

 

Debian LTS Work February 2016

This was my tenth month as a Freexian sponsored LTS contributor. I was assigned 8 hours for the month of February.

As I did last month, I worked on updating clamav in wheezy and squeeze-lts.  As with previous updates to clamav, we updated it to the new upstream version[1].  As an added complexity, this version bumped soname, so it’s now libclamav7 instead of libclamav6.  This bump necessitated a small transition in jessie/wheezy-proposed-updates and squeeze-lts.

The update for Jessie (included for completeness here) was done early in the month by other pkg-clamav team members.  It and the rebuilt/update libclamav reverse-depends will be included in the next Jessie point release.

For wheezy, I uploaded libclamunrar (which bumped soname as well) and worked with other pkg-clamav team members on getting clamav to build on sparc and preparing a fix for c-icap.  It and the rebuilt/update libclamav reverse-depends will be included in the next Wheezy point release.

As a result of the amount of time it took, the squeeze-lts update landed later than I hoped it would, but it is there.  As documented in DLA 437-1, there are new packages for clamav, libclamunrar, python-clamav, and klamav.  The last squeeze libclamav reverse-depend, dansguardian, took more work, but it too is updated, see DLA 440-1.

 

[1] The primary reason for this is that anti-virus is an arms race.  Unlike other types of packages being stable with only fixes for severe bugs and security issues does not result in a stable capability.  It will regress over time.  In order to keep up, the new version is needed.

Postfix 3.0 woes

Postfix 3.0 recently hit Debian Unstable (and Ubuntu Xenial for those that care about that).  It’s been a bit of a bumpy road, but it seems to mostly be there for new installs.  For package upgrades, there’s still issues.  We hope to have that sorted shortly, but in the meantime, all you should need to do to get an upgraded system working is add or adjust two parameters in your main.cf

shlib_directory=/usr/lib/postfix
daemon_directory=/usr/lib/postfix/sbin

You can either edit the file directly or use postconf:

postconf -e shlib_directory=/usr/lib/postfix
postconf -e daemon_directory=/usr/lib/postfix/sbin

No need to file more bugs and yes, we also know postfix 3.1 was just released.  One thing at a time.

Debian LTS Work January 2016

This was my ninth month as a Freexian sponsored LTS contributor. I was assigned 8 hours for the month of January.

My time this month was spent preparing updates for clamav and the associated libclamunrar for squeeze and wheezy.  For wheezy, I’ve only helped a little, mostly I worked on squeeze.

This update is more complex than usual because with clamav 0.99 upstream bumped soname and so in addition to the normal case of transitions in unstable, we’ve needed transitions for stable, oldstable, and squeeze-lts.  We also try to be careful and maintain higher versions in newer releases, so stable needed to wait for 0.99 in testing, oldstable needed to wait for stable, etc.

Currently 0.99 is in stable proposed updates and I’ve requested that the update for wheezy (oldstable) go forward.  Once that’s done, I’ve got squeeze-lts ready to go.

Python Packaging Build-Depends

As a follow-up to my last post where I discussed common Python packaging related errors, I thought it would be worth to have a separate post on how to decide on build-depends for Python (and Python3) packages.

The python ecosystem has a lot of packages built around supporting multiple versions of python (really python3 now) in parallel.  I’m going to limit this post to packages you might need to build-depend on directly.

Python (2)

Since Jessie (Debian 8), python2.7 has been the only supported python version.  For development of Stretch and backports to Jessie there is no need to worry about multiple python versions.  As a result, several ‘all’ packages are (and will continue to be) equivalent to their non-‘all’ counterparts.  We well continue to provide the ‘all’ packages for backward compatibility, but they aren’t really needed any more.

python (or python-all)

This is the package to build-depend on if your package is pure Python (no C extensions) and does not for some other reason need access to the Python header files (there are a handful of packages this latter caveat applies to, if you don’t know if it applies to your package, it almost certainly doesn’t).

You should also build-depend on dh-python.  It was originally shipped as part of the python package (and there is still an old version provided), but to get the most current code with new bug fixes and features, build-depend on dh-python.

python-dev (or python-all-dev)

If your package contains compiled C or C++ extensions, this package either provides or depends on the packages that provide all the header files you need.

Do not also build-depend on python.  python-dev depends on it and it is just an unneeded redundancy.

python-dbg (or python-all-dbg)

Add this if you build a -dbg package (not needed for -dbgsym).

Other python packages

There is not, AFAICT, any reason to build-dep on any of the other packages provided (e.g. libpython-dev).  It is common to see things like python-all, python, python-dev, libpython-dev in build-depends.  This could be simplified just to python-all-dev since it will pull the rest in.

Python3

Build-depends selection for Python 3 is generally similar, except that we continue to want to be able to support multiple python3 versions (as we currently support python3.4 and python3.5).  There are a few differences:

All or not -all

Python3 transitions are much easier when C extensions are compiled for all supported versions.  In many cases all that’s needed if you use pybuild is to build-depend on python3-all-dev.  While this is preferred, in many cases this would be technically challenging and not worth the trouble.  This is mostly true for python3 based applications.

Python3-all is mostly useful for running test suites against all supported python3 versions.

Transitions

As mentioned in the python section above, build-depends on python3-{all-}dev is generally only needed for compiled C extensions.  For python3 these are also the packages that need to be rebuilt for a transition.  Please avoid -dev build-depends whenever possible for non-compiled packages.  Please keep your packages that do need rebuilding binNMU safe.

Transitions happen in three stages:

1. A new python3 version is added to supported python3 versions and packages that need rebuilding due to compiled code and that support multiple versions are binNMUed to add support for the new version.
2. The default python3 is changed to be the new version and packages that only support a single python3 version are rebuilt.
3. The old python3 version is dropped from supported versions and packages will multiple-version support are binNMUed to remove support for the dropped version.

This may seem complex (OK, it is a bit), but it enables a seamless transition for packages with multi-version support since they always support the default version.  For packages that only support a single version there is an inevitable period when they go uninstallable once the default version has changed and until they can be rebuilt with the new default.

Specific version requirements

Please don’t build-depend against specific python3 versions.  Those don’t show up in the transition tracker.  Use X-Python3-Version (see python policy for details) to specify the version you need.

Summary

Please check your packages and only build-depend on the -dev packages when you need it.  Check for redundancy and remove it.  Try and build for all python3 versions.  Don’t build-depend on specific python3 versions.

Python3.5 is default python3 in sid

As of today, python3 -> python3.5.  There’s a bit of a transition, but fortunately because most extensions are packaged to build for all supported python3 versions, we started this transition at about 80% done.  Thank you do the maintainers that have done that.  It makes these transitions much smoother.

As part of getting ready for this transition, I reviewed all the packages that needed to be rebuilt for this stage of the transition to python3.5 and a few common errors stood out:

1. For python3 it’s {python3:Depends} not {python:Depends}.
2. Do not use {python3:Provides}.  This has never been used for python3 (go read the policy if you doubt me [1]).
3. Almost for sure do not use {python:Provides}.  The only time it should still be used is if some package depends on python2.7-$PACKAGE. It would surprise me if any of these are left in the archive.  If so, since python2.7 is the last python2, then they should be adjusted.  Work with the maintainer of such an rdepend and once it’s removed, then drop the provides.
4. Do not use XB-Python-Version.  We no longer use this to manage transitions (there won’t be any more python transitions).
5. Do not use XB-Python3-Version.  This was never used.

Now that we have robust transition trackers [2], the purpose for which XB-Python-Version is obsolete.

In other news, pysupport was recently removed from the archive.  This means that, following the previous removal of pycentral, we finally have one and only one python packaging helper (dh-python) that supports both python and python3.  Thanks to everyone who made that possible.

 

[1] https://www.debian.org/doc/packaging-manuals/python-policy/

[2] https://release.debian.org/transitions/html/python3.5.html