As far as I could tell, these are the most commonly occurring words in titles of papers listed in MathSciNet, excluding the likes of "and", "of", "the", etc.:
equation (169,596), problem (152,343), system (123,957), function (112,503), space (102,400), solution (98,857), theory (97,817), method (82,817), group (73,474), linear (66,798), theorem (65,826), differential (65,421), model (60,637), operator (56,991), class (51,260), two (48,300), order (48,261), application (46,494), boundary (44,574), finite (44,135), field (43,788), type (41,569), integral (41,114), property (39,699), time (38,130), set (37,639), point (36,067), analysis (35,658), value (34,789), matrix (34,774), approximation (34,520), certain (34,010), control (32,073), number (31,463), algorithm (31,164), distribution (30,665), polynomial (29,614), stability (29,257), graph (29,178), asymptotic (28,532), condition (28,100), note (28,062), representation (27,766), random (26,742), quantum (25,396), wave (24,922), optimal (24,741), series (24,704), variable (24,351), ring (22,547), stochastic (21,701), flow (21,365), dynamic (21,362), inequality (21,050), form (20,680), existence (20,414), invariant (20,334);
with words such as algebraic, behavio(u)r, bounded, characteristic, coefficient, complex, construction, continuous, curve, data, decomposition, difference, elliptic, extension, first, fixed, fluid, free, fuzzy, global, hyperbolic, infinite, inverse, large, limit, local, map, mapping, mean, mechanics, metric, module, motion, normal, numerical, one, parabolic, partial, periodic, positive, potential, principle, product, real, regular, relation, remark, second, sequence, simple, singular, spectral, sum, symmetric, three, transformation, vector and zero following on their heels.
A few more surprising words also appear, and not all in papers by statisticians performing unlikely analyses:
sandwich (224), bandit (206), monster (116), sex (116), secretary (110), god (56), sausage (53), devil (50), cake (42), egg (39), demon (38), football (31), snark (29), worm (25), tadpole (24), spider (23), frog (21), hairy (20), ham (19), mouse/mice (18), sober (16), sponge (13), apple (12), banana (11), baseball (11), cheese (11), tortoise (10), dirty (9), slug (8), cricket (7), coffee (6), lobster (6), blotto (4), murder (4), snail (4), penguin (3), tea (3), vampire (3), boojum (2), drunkard (2), merry-go-round (2), mole (2), butter (1), grebe (1), jazz (1), molehill (1), python (1), silly (1), sloth (1), spam (1), whelk (1), ...
(Recall that there are concepts known as the Ham Sandwich theorem, the Monster group, the Wiener sausage, the Devil's staircase, the science of cake cutting, egg domains, Maxwell's demon, snarks (graph theory and, independently, topology), dilaton tadpoles (mathematical physics), leap-frog algorithms, the Hairy Ball theorem, mice (set theory), sober spaces, Swiss cheeses, the Achilles and tortoise paradox, dirty black holes, slug flow, lobster trees, Colonel Blotto's game, the drunkard's walk, and the problem of logical sloth.)
Here are a few words which do not yet occur in titles of mathematical papers, and are thus a challenge to future mathematicians:
albatross, custard, disgusting, elk, foolish, jackdaw, lumberjack, puffin, raving, weasel, wombat.
Jonathan Partington, last updated January 23rd 2003. Thanks due to various contributors (you know who you are).