An update on the Fedora August 2008 intrusion
From: | "Paul W. Frields" <stickster-AT-gmail.com> | |
To: | fedora-announce-list <fedora-announce-list-AT-redhat.com> | |
Subject: | Update and Report on Fedora August 2008 Intrusion | |
Date: | Mon, 30 Mar 2009 10:00:15 -0400 | |
Message-ID: | <20090330140015.GG21944@localhost.localdomain> |
This communication provides additional information on the Fedora infrastructure intrusion first reported on August 14, 2008. In part this communication reiterates information provided in previous announcements. On August 12, 2008, a failed cron job on a Fedora host reported an error to the Fedora system administrators. While investigating the source of this error, the sysadmins reviewed the recent logs and discovered that the package complement on the host had changed. Further investigation showed that changes were the result of tampering by an intruder. Once the extent of the problem was discovered, we notified the community: https://www.redhat.com/archives/fedora-announce-list/2008... The compromise was not the result of a software vulnerability, and as we have previously stated, our investigation has revealed no such vulnerabilities. Instead, the intruder took a copy of a SSH private key which was not secured with a passphrase from a system outside the Fedora infrastructure. The intruder then used that key, which belonged to a Fedora administrator, to access Fedora systems. In addition, based on system log sudo entries, we believe the intruder also compromised that user's password. New security policies for Fedora administrators directly address the further safeguarding of keys, as noted below. The Fedora package signing key was present on a system to which the intruder had access during the time of the event, but the results of our investigation did not lead us to believe the intruder accessed the key. The intruder used the account's privileges to build modified versions of openssh and rpm. These packages would have allowed the intruder to capture the passphrases of unwitting users on our build system, or the passphrase used to secure the package signing key. With the signing key's passphrase and access to that key, the intruder would have been able to fraudulently sign packages. The intruder did deploy the modified packages, and the modified SSH package may have captured passphrases for a short time. However, the investigation supports the conclusion that the modified packages were discovered before anyone accessed the system to sign any packages using the modified RPM package. Therefore, we do not believe the intruder ever had access to the passphrase to the signing key. Nevertheless, previous announcements have explained clearly the precautionary reasons for our earlier decision to deploy a new package signing key. Details on the process and progress of the signing key changes can be found here: https://fedoraproject.org/wiki/New_signing_key The modified packages were only installed on a small number of Fedora infrastructure systems, and therefore we do not believe Fedora account holders were at significant risk during the time the intruder had these packages in place. We also required passphrase and SSH key changes from Fedora users, to increase our confidence going forward. To reiterate, our analysis supports our initial findings that the Fedora Project infrastructure delivered no software compromised by the intruder to any of its mirrors, or the master repository from which they synchronize content. Our investigation also shows that the intrusion affected only a few internal Fedora infrastructure servers. Most of the mitigation work done by the Fedora Infrastructure team was precautionary, and allows us to have higher confidence in our present and future work. The Fedora Infrastructure team quickly responded to the threat, isolated the systems in question from the network, made snapshots of affected hosts, assessed the damage, and proceeded to rebuild Fedora's entire infrastructure essentially from scratch. By the end of the first week after the event, most essential systems were functioning nominally again. The remainder of the work took close to three weeks to complete fully, and return all systems to normal status. To increase our security posture, the Fedora Infrastructure team now requires all members of the Fedora system administrator groups to use passphrases on their private keys. That policy is found, along with other policies of the Infrastructure team, here: http://infrastructure.fedoraproject.org/csi/security-policy/ The Infrastructure team is also actively developing a Community Services Infrastructure project (http://fedorahosted.org/csi) and supporting documentation to include these lessons in their guidance. Team members have also worked on deploying SELinux throughout our infrastructure as well as an audit and prelude intrusion detection system for added good measure. Furthermore, the Fedora release engineering team is also leading efforts on further improvements and refinements to Fedora's package signing system and procedures. Community members are invited to contact the appropriate Fedora team if they wish to get involved in any of these processes. The specific facts that emerged during the investigation, which was international in nature, made it necessary to restrict information flowing to the public, to avoid damaging the effectiveness of the investigation. That restriction has been unfortunate, and is not in the ordinary spirit of openness and transparency with which we strive to define the Fedora Project. We appreciate the patience and support the Fedora community has given us over the past several months. We also wish to express our thanks for the technical assistance of the security response team at Red Hat. This report concludes the matter from the Fedora Project's perspective, and provides a final accounting of the intrusion event and response by the Fedora infrastructure and management teams. Following is a detailed time line, in UTC, of activities by Fedora contributors. = = = = = 2008-08-12 01:00:00 - Last packaging signing process from a Fedora admin. Key would have been on host temporarily up until this time. 2008-08-12 07:49:05 - Standard Fedora 'pkgconfig' package installed by the intruder. This package is required to build an 'openssh' package. Intruder proceeds to build a modified 'openssh' package. 2008-08-12 08:10:46 - modified 'openssh' package installed by intruder. 2008-08-12 17:46:50 - Standard Fedora 'gettext' package installed by intruder. This package is required to build an 'rpm' package. 2008-08-12 20:18:36 - Standard Fedora 'mc' package installed by intruder, possibly for convenience of stealth. 2008-08-12 21:33:59 - Bacula backup started (scheduled job) 2008-08-12 22:01:54 - Bacula backup Ended 2008-08-12 22:31:51 - modified 'rpm' package installed (along with standard Fedora package dependencies for 'rpm'). 2008-08-12 22:51:00 - Cron job failed, notified admins. 2008-08-12 22:53:00 - Fedora Infrastructure admins first noticed and started poking around at why RPM had changed. 2008-08-12 23:11:00 - Infrastructure team lead is notified and more prodding begins. 2008-08-12 23:38:00 - Infrastructure team members gather for discussions on dedicated, private IRC channel and conference call. 2008-08-13 01:50:00 - It becomes more clear that a script is not at fault. LVM snapshot taken. 2008-08-13 04:00:14 - Bacula backup (during the intrusion) restored to secure location 2008-08-13 04:04:14 - Discovery of an RPM in /root/.ssh/ provides proof of malicious intent. 2008-08-13 04:05:00 - Red Hat security team notified. 2008-08-13 04:46:00 - Compromised host prohibited from routing out or in. All machines on its network are preventing access from it. Outbound connections logged. 2008-08-13 05:16:00 - Fedora Project Leader notified. 2008-08-13 06:13:00 - Host state saved (Xen guest). We have a running copy of the host as it was without a reboot. 2008-08-13 06:14:00 - Users who have accessed the machine during the intrusion advised to change their passwords and SSH keys. 2008-08-13 10:13:00 - Work continues in concert with Red Hat security team members. Preliminary announcement prepared 2008-08-14 17:36:00 - All passwords and SSH keys disabled. 2008-08-14 23:15:13 - Preliminary announcement to fedora-announce-list, 1+19:11 after initial determination of malicious event. 2008-08-15 02:47:00 - All administrator access forced to shell access only for partial re-enabling of account system. 2008-08-15 12:00:00 - (approximate) Fedora's package build system, koji, patched to revoke all access. 2008-08-15 13:11:00 - Last package build routine allowed to complete before shutdown. Comprehensive verification of the build system database contents begins, comparing against known source for malicious content. 2008-08-16 15:30:03 - Update announcement to fedora-announce-list, 3+11:26 after initial determination of malicious event. 2008-08-17 22:34:00 - Members of sysadmin-web group allowed back on app servers. 2008-08-18 04:06:31 - Primary content verification of build system and CVS completed. 2008-08-18 18:06:00 - CVS admins allowed back on servers, and handle additional verification for hosted projects. 2008-08-19 02:07:45 - Update announcement to fedora-announce-list, 5+22:03 after initial determination of malicious event. 2008-08-19 02:37:00 - Hosted project verification completed, and Fedora Hosted back online. 2008-08-19 20:19:00 - Anonymous access via cvspserver allowed. 2008-08-20 02:53:00 - Writable access to cvs1 reactivated. 2008-08-20 18:35:00 - Koji build system officially open and building again. 2008-08-22 12:00:02 - Update announcement to fedora-announce-list, 9+07:56 after initial determination of malicious event. 2008-09-19 02:41:29 - Update announcement to fedora-announce-list, 37+22:37 after initial determination of malicious event. Investigation and issue resolution continues. 2009-03-30 14:00:00 - Final report to fedora-announce-list, 229+9:56 after initial determination of malicious event. = = = = = -- Paul W. Frields http://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug -- fedora-announce-list mailing list fedora-announce-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-announce-list
(Log in to post comments)
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 14:24 UTC (Mon) by JoeBuck (subscriber, #2330) [Link]
Why the long delay in reporting this? Did it have something to do with an attempt to prosecute the intruder?
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 14:27 UTC (Mon) by kragil (guest, #34373) [Link]
Although not secured private keys seem like a very obvious hole in any security policy ..
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 14:46 UTC (Mon) by skvidal (guest, #3094) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 14:47 UTC (Mon) by mmcgrath (guest, #44906) [Link]
Yeah, we kind of just assumed people would be doing that. No longer though, the new policy is pretty explicit about password protecting keys :)
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 15:11 UTC (Mon) by melevittfl (guest, #5409) [Link]
passwords on the public keys rather than rely on people following a policy document?
I can't imagine any way to do this with 100% effectiveness because you'd have to trust the client
side to tell you the truth and a determined policy violator could simply build a custom ssh client
that would lie to the server.
Never mind... ;)
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 15:34 UTC (Mon) by iabervon (subscriber, #722) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 16:13 UTC (Mon) by bfields (subscriber, #19510) [Link]
a determined policy violator could simply build a custom ssh client that would lie to the server.
Well, you're not trying to defend against malicious users--you've already trusted them with access--you're just trying to remind them of policy and protect them against their own mistakes.
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 17:48 UTC (Mon) by wmf (guest, #33791) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 31, 2009 2:31 UTC (Tue) by motk (subscriber, #51120) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 18:17 UTC (Mon) by proski (subscriber, #104) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 31, 2009 9:19 UTC (Tue) by muwlgr (guest, #35359) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 19:28 UTC (Mon) by drag (guest, #31333) [Link]
Nope.
> I can't imagine any way to do this with 100% effectiveness because you'd have to trust the client side to tell you the truth and a determined policy violator could simply build a custom ssh client that would lie to the server.
Ya. Your right.
This is much much better for large orginizations to disable public key authorization support and use Kerberos instead.
An update on the Fedora August 2008 intrusion
Posted Apr 5, 2009 10:42 UTC (Sun) by dlang (guest, #313) [Link]
also, there are good reasons to not use passwords sometimes. think of cases where you want automated processes to ssh into other systems. there is no user around to type the password in.
An update on the Fedora August 2008 intrusion
Posted Apr 5, 2009 12:01 UTC (Sun) by nix (subscriber, #2304) [Link]
use passphrases, thanks to the existence of ssh-agent.
An update on the Fedora August 2008 intrusion
Posted Apr 5, 2009 12:10 UTC (Sun) by dlang (guest, #313) [Link]
An update on the Fedora August 2008 intrusion
Posted Apr 6, 2009 2:29 UTC (Mon) by knobunc (guest, #4678) [Link]
http://www.enterprisenetworkingplanet.com/netsecur/articl...
(Keychain is wonderful)
-ben
An update on the Fedora August 2008 intrusion
Posted Apr 6, 2009 8:22 UTC (Mon) by dlang (guest, #313) [Link]
so you end up with either setting up a key that doesn't have a passphrase, or having to store that passphrase in a script (or a bunch of scripts since they don't all run as part of a single user session)
I don't see a big win in security to counter the extra complexity here.
no, this isn't appropriate for cases like what was involved in the Fedora intrusion, but the claim was made (several posts up) that there is no legitimate reason to have a blank passphrase, and that is what I'm disputing.
An update on the Fedora August 2008 intrusion
Posted Apr 6, 2009 12:44 UTC (Mon) by knobunc (guest, #4678) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 15:41 UTC (Mon) by qg6te2 (guest, #52587) [Link]
Yes, a quick response, but the detection of the intrusion can be considered as a fluke. The margin for error appears to have been too little to begin with.
2008-08-12 22:51:00 - Cron job failed, notified admins.
If the intruder had been just a little bit more careful, it is entirely possible that either the security problem would have been an order of a magnitude bigger when finally discovered (e.g. many compromised packages, with possible destructive payloads, botnet operation, etc), or to this day we would have no idea that there is something is going on.
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 15:49 UTC (Mon) by spot (guest, #15640) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 16:14 UTC (Mon) by kragil (guest, #34373) [Link]
It always takes events like this to make changes .. I really hope other distros learn from your mistakeS.
Would be great if Novell, Canonical and Debian would openly disclose how their infrastructure is protected. (In detail.)
At the end of the day my system is dependent on a secure infrastructure from supplier of the binary packages.
Security by obscurity does not work .. let people openly discuss the measures taken and you will deter attackers and possibly strengthen your system by getting new ideas or by finding (obvious) flaws.
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 19:03 UTC (Mon) by nix (subscriber, #2304) [Link]
failed cron job would have been no clue. (Of course that may have been
hard: he apparently hadn't escalated to root on that machine, let alone
got at the loghost(s)...)
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 16:09 UTC (Mon) by tialaramex (subscriber, #21167) [Link]
An update on the Fedora August 2008 intrusion
Posted Mar 30, 2009 23:35 UTC (Mon) by gdt (subscriber, #6284) [Link]
There's a wide variety of choices. Challenge-response cards being one obvious alternative. Or patching sshd so that authentication can be stacked (eg: key followed by password).
The blocking issue is a lack of federated single sign on. You can make authentication more complex but people aren't willing to go through multi factor rigmarole for every resource accessed [as this episode nicely demonstrates]. SSO limits the number of times authentication is requested to a few times a day.
An update on the Fedora August 2008 intrusion
Posted Mar 31, 2009 12:33 UTC (Tue) by tialaramex (subscriber, #21167) [Link]
If this Fedora contributor ran Fedora, they had the option to enter their SSH passphrase as infrequently as once per (desktop) login. Is that too much?
An update on the Fedora August 2008 intrusion
Posted Mar 31, 2009 17:04 UTC (Tue) by chaneau (guest, #6674) [Link]
If this Fedora contributor ran Fedora, they had the option to enter their SSH passphrase as infrequently as once per (desktop) login. Is that too much?
That's the missing answer from this report is it not? How did the intruder gain access to the private key in the first place?
Did the intruder have physical access?
Did he access the key remotely?
Did the Fedora guy leave his key on some untrusted computer?
Was the computer stolen?
Some of these questions are more frightening than the others, but if you want me to trust Fedora, the quality and the seriousness of it's administrators, they should tell us what really happened
An update on the Fedora August 2008 intrusion
Posted Apr 9, 2009 18:44 UTC (Thu) by eric.rannaud (guest, #44292) [Link]
How was the private key first acquired?