ActiveX controls are Microsoft's equivalent of Java applets. Unlike Java, they don't run in a sandbox, and have unlimited access to your computer. The security implications of that statement were not immediately apparent. Microsoft have been endeavoring to make ActiveX secure, first with SP2 for XP, and later with IE7. The truth is that this is an attempt to impose security on inherently insecure content, and that ActiveX will not be entirely secure until IE7 in Vista, when it's activity will be strictly confined to the browser.

Security

ActiveX was never designed with security in mind:

"Microsoft recognized the problem with ActiveX as far back as 1996 when Charles Fitzgerald, program manager of Microsoft's Java team said 'If you want security on the 'Net', unplug your computer. ... We never made the claim up front that ActiveX is intrinsically secure.'" Wikipedia

ActiveX controls are potentially very harmful to your computer:

"...an ActiveX control is essentially a Windows program that can be distributed from a web page. These controls can do literally anything a Windows program can do. That means you could write an ActiveX control to erase a hard drive. A control containing a virus or trojan can be written, distributed, and activated from a web page, and the viewer of the control might never know. A control could even scan your drive for tax records or documents the control's author was interested in, and e-mail them off to some other person. All this can be done in a control that pretends to be something interesting, like a video game."

"Why is ActiveX dangerous?"  halcyon.com

Security for ActiveX was intended to be provided by the controls being signed as safe by their author, but flaws in this system soon became obvious. Lax security settings could allow even unsigned ActiveX controls to be installed from a web site, and even signed controls were not always trustworthy:

"Code Signing simply attempts to identify who signed the control. Anyone can go out and get a code signature. It's a pretty much automatic process. You go to a web site, give them a name, address, credit card number and some other stuff (none of which have to be yours), click "I Agree" on a page full of legal jargon, and pretty soon you get an e-mail with the information you need to sign the control in it. Once you have your Digital ID, you can sign any unsigned ActiveX control. Nobody reviews these controls! In other words, a signature doesn't tell you who wrote the control and it doesn't tell you if the control is safe or not. Heck, with the number of hot credit card numbers out on the net, it doesn't even tell you for sure who signed it. A danger is that seeing that a control is signed will give folks a warm fuzzy feeling about the control, and encourage them to run it, even though it does not guarantee their safety! "

"Why is ActiveX dangerous?"  halcyon.com

There were other problems with the original system: ActiveX certificates issued to signed controls could be misleading or deceptive, and Windows allowed repeated pop-ups requesting permission to install the control.

How VeriSign Could Stop Drive-By Downloads benedelman.org
"Signed ActiveX apps are certified, but can still contain malicious code!" safer-networking.org

Security was tightened up in Windows XP SP2, with unsigned controls no longer allowed to be installed by web sites, and a new security bar warning of the security risks and requiring several user confirmations (and not allowing multiple pop-ups) before allowing the user to install an ActiveX control.

This still left the opportunity for 'social engineering'- malicious sites soon started illustrating how to install ActiveX controls by clicking through the various confirmation stages.

And another problem with ActiveX remained: legitimate ActiveX controls placed on the computer by web sites or there as part of the operating system could be exploited to take control of the system if a vulnerability was found.

"An ActiveX control can be an extremely insecure way to provide a feature. Because it is a Component Object Model (COM) object, it can do anything the user can do from that computer. It can read from and write to the registry, and it has access to the local file system. From the moment a user downloads an ActiveX control, the control may be vulnerable to attack because any Web application on the Internet can repurpose it, that is, use the control for its own ends whether sincere or malicious." MSDN

See also:

Research: Buggy, Flawed 'ActiveX' Controls Pervasive washingtonpost.com

At the original time of writing of this page (the end of 2006), there had been several recent security vulnerabilities in Internet Explorer in which ActiveX vulnerabilities allowed installation of malware without the user being required to accept the installation.

Internet Explorer DHTML Edit ActiveX Control Cross-Site Scripting Secunia
WebViewFolderIcon setSlice exploit in the wild - follow up Exploit Prevention Labs
Another zero day on the loose? keyframe (daxctle.ocx) exploit seen in the wild Sunbelt Blog
Critical Microsoft Windows 0-day appears Security Focus

IE7 on XP locks down many ActiveX controls so that web sites cannot exploit any vulnerabilities found, but some controls remain accessible, because they are deemed secure. At the time of writing, malware was exploiting a flaw in such a control:

Attackers end-run around IE security Security Focus

Update: Noticing that I hadn't updated this page for a year, I decided to take a look at ActiveX security one year later. Here's what I found:

"During the past year, hundreds of vulnerabilities in ActiveX controls installed by Microsoft and other software vendors have been discovered. These are also being exploited via Internet Explorer."

SANS Top-20 2007 Security Risks (2007 Annual Update) www2.sans.org

"According to the Symantec Internet Security Threat Report, in the first half of 2007 Symantec documented 237 vulnerabilities affecting browser plug-ins. Vulnerabilities affecting ActiveX components comprised 210 of the 237 issues. This represents an increase of 167 more issues; or, over five times the amount of vulnerabilities released during the last half of 2006.

Interestingly (or perhaps more disturbingly) there has also been a rise in proof-of-concept and exploit code that has been made available for ActiveX vulnerabilities released over the past year and a half."

On the day of writing this update, there were two ActiveX vulnerabilities in the news:

Zero-day flaw haunts HP laptop models blogs.zdnet.com
Security Advisory 0703 videolan.org

So it looks like AcitveX remains a security worry. At least in 2007 there are no stories of zero-day ActiveX exploits in the wild. The exploit code "being exploited via Internet Explorer" affects out of date versions of the browser, and patches were made available before attacks occurred. Still, the stories highlight the need to keep IE and Windows programs which use ActiveX up to date to avoid security risks. (End of 2007 update.)

In Vista, Microsoft will final make ActiveX secure:

Under Vista, Internet Explorer will run in a new mode that limits access to most of the operating system's facilities, protecting data and the software. Called "protected mode," the feature will limit the access rights of Internet Explorer and stop most ActiveX abuse in its tracks.

In effect, Internet Explorer will be sandboxed, at last providing the sort of "bottom-up" security for ActiveX- the sort of security that Java had from the beginning, and the lack of which in ActiveX has lead to so many problems.

See also:

ActiveX in Windows- What is it? surfthenetsafely.com

What are the risks?

An ActiveX control has complete access the system, so installing an ActiveX control is potentially as dangerous as clicking on an executable file: you could be installing a Trojan horse to steal personal information or download more malware, an adware program to bombard you with pop-ups, or a premium rate dialler to charge you extortionate rates to connect to the internet.

Users of Internet Explorer on systems not updated to XP SP2 a vulnerable to persistent and deceptive ActiveX installation requests.This is usually achieved by presenting a sequence of dialogue boxes which may be misleading labelled or seemingly impossible to get rid of without accepting. If the ActiveX software is accepted, it installs a variety of spyware.

Users of Internet Explorer on XP SP2 are still vulnerable to 'social engineering' attacks, where malicious web sites invite viewers to believe ActiveX controls are harmless, suggest they ignore security warnings, and illustrate how to install a control. (To be fair, similar attacks are also possible with Java.)

Users of Internet Explorer 6 are vulnerable to drive-by malware downloads- where malware is installed simply by visiting a malicious web site, without the user being required to accept an installation- if a vulnerability is found in any ActiveX control present on their computer. At the time of writing, such a vulnerability was being actively exploited. Research suggests more such vulnerabilities may emerge:

ActiveX security faces storm before calm Security Focus

Users of Internet Explorer 7 are vulnerable to drive-by malware downloads if a vulnerability is found in a more limited number of "secure" ActiveX controls present on all computers. At the time of writing, such a vulnerability was being actively exploited.

Malicious ActiveX controls have been used to install premium-rate diallers, costing many people on dial up connections many hundreds of pounds when their internet connection was diverted without their knowledge to a premium rate telephone number.

What should I do?

Consider a browser other than Internet Explorer. Other browsers do not support ActiveX and are not vulnerable. You will have to keep IE on your computer to access sites which use ActiveX, for example the Microsoft Update site.

Why You Should Dump Internet Explorer lockergnome.com
US-CERT: Beware of IE internetnews.com
Mozilla Firefox 1.0 ZDNet UK

Windows XP SP2 has more control of what ActiveX controls are loaded, giving you more protection against malicious programs loading without your consent.

IE7 locks down ActiveX controls already on the computer, and is an essential security upgrade, even if it is not your default browser.

Protect yourself against malicious ActiveX controls.

Block bad sites

Sites added to the Internet Explorer restricted Zone cannot use ActiveX controls, signed or unsigned. Please see the Blocking Bad Sites page to learn how to do this.

Block Malicious ActiveX controls

Each ActiveX control has a unique identifying code, and it is possible to add a special 'killbit' to the Windows registry to block a malicious ActiveX control. An ActiveX control which has been used to hijack a computer or invade privacy will then not be able to run.

How to Stop an ActiveX Control from Running in Internet Explorer support.microsoft.com

Here are two ways of protecting yourself against a number of spyware ActiveX controls:

Spyware Blaster: "Developed in 2002 to help users prevent spyware installs, Spyware Blaster pioneered the effective use of prevention techniques to reduce or greatly eliminate spyware-related problems, as well as problems related to other unwanted software such as dialers, browser hijackers, and adware."

Javacool Software's Spyware Blaster

Spyware Block List File: "Tired of all that Spyware and Adware crap being installed by ActiveX ? But don't want to lose out on functionality? We have created a system that blocks all known "bad" ActiveX controls from running inside Internet Explorer by setting the "Kill bit"."
Spyware Guide's Spyware Block List File

Spybot Search and Destroy also offers an 'immunization' feature to block bad ActiveX controls.

Unsigned ActiveX controls are blocked by default in Internet Explorer. Make sure you don't have the slider set any lower than medium for the Internet Zone.