The development of consonant gradation in Finnic (and why not, also elsewhere in Uralic) is one of those topics that really needs a new monograph-scale treatment one of these days. Not just for the sake of collecting the accumulated knowledge in a single source, either. Modern understanding of linguistic theory and methodology would probably allow not only improoved description of gradation as it works in the modern languages; it should also help better tackling the historical puzzles involved.
Some overall observations are simple enough to make. Perhaps the main historical trend in Finnic has been the gradual morphologization of gradation, departing from its original phonetic roots, and being generalized in some environments, levelled in others.
One good example is the gradation of /t/ in consonant clusters: we can reconstruct *nd, *ld, *rd as the original weak grades of the sonorant-initial clusters *nt, *lt, *rt (exactly parallel to *d as the weak grade of intervocalic *t); however, after further phonetic development, most Finnic varieties now rather have /nn/, /ll/, /rr/. Given the pattern consonant+t : geminate consonant here, it is not too surprizing that large swaths of Finnic varieties have also introduced the parallel alternation /st/ : /ss/, even though the weak grade clearly cannot originate from earlier **sd.  The innovation covers Karelian proper (but not Livvi/Olonetsian); Ingrian; and various eastern dialects of Estonian. Finnish dialects, though, have no trace of this.
— A small example for readers who are not especially familiar with how Finnic root-medial consonant gradation works in practice: the verb roots *anta- ‘to give’, *osta- ‘to buy’ yield in Karelian the 1st person singular forms annan ‘I give’, oššan ‘I buy’, with /nn/ and /šš/ as the weak-grade forms of /nt/, /št/. Finnish by contrast has similarly weak-grade annan, but a “strong-grade” form (rather: unaffected by gradation) ostan. In both languages, the underlying cluster also remains e.g. in the infinitive forms: Krl. antoa, oštoa; Fi. antaa, ostaa.
Which morphological forms show gradation can be predicted from the Proto-Finnic syllable structure. 1PS *andan, *ostan have a closed 2nd syllable, triggering lenition of *t, while the infinitives *antadak, *ostadak have an open 2nd syllable, and the original voiceless stop remains. However, as we can see, in which exact phonetic environments /t/ is affected by gradation varies by language. In this case gradation appears more heavily morphophonologized in Karelian than in Finnish; both languages however retain the original phonetic conditioning factors of gradation (a closed 2nd syllable in annan, vs. an open one in antoa/antaa), and hence these cases of gradation could be still called morphophonological, not yet purely morphological.
Back on track. Interestingly, in the middle of the above-mentioned innovative area where original *st has been subjected to gradation, also another development is found: in Votic, *st is reflected as /ss/ in the strong grade, and single /s/ in the weak grade (thus e.g.: *ostada > õssaa, *ostan > õsaa). Traditionally this has been attributed to an early separate development: a general sound change *st > *ss would have taken place already before the analogical extension of *Ct-gradation. This would have been then followed instead by the analogical extension of the gradation pattern geminate voiceless stop : singleton voiceless stop also to the voiceless fricative /s/, now abundantly found as a geminate as well.
Looking at Votic in isolation, this seems like an entirely possible account. However, in an areal context this is less clear. Votic is the most persistently innovative Finnic language with respect to consonant gradation: it is applied productively even to recent loanword consonants and clusters from Russian (resulting in such unique alternations as /pk/ : /bg/). In this light, it seems like an unusual coincidence that, at the epicenter of the eventual /st/-gradating area,  Votic would have already early on been established as an island that opted for a different solution altogether. An alternate hypothesis would be to suppose that Votic once used to have the more widespread pattern *st : *ss as well; and that the attested pattern /ss/ : /s/ represents simply a further development for this. This is fairly easy to arrange. We can continue to assume *st > /ss/ as a regular sound law, and would have to merely combine it with also assuming *ss > /s/, as a mini-chainshift of sorts.
Not everyone will like this kind of complication just for the sake of neater geographical generalizations, I’m sure. But there’s an interesting piece of evidence that appears to be in favor of my new analysis. The inessive case ending, reconstructible as Proto-Finnic *-ssA,  is in fact found in Votic as -za ~ -zä. This also suggests exactly the development *ss > *s, followed by voicing between two unstressed syllables (— also known as “suffixal gradation”, though the process differs from regular gradation in a number of ways).
This seems to be all the evidence we can hope to get together on the development of *ss in Votic, though. As far as I know, there are no root-medial cases of *ss that could be reconstructed for Proto-Finnic, and not even any especially old loanwords. The most widespread might be the obviously recent Fi/Krl. pyssy ~ Es. püss ~ Vo. püssü ‘gun’ (from Low German).
It’s again possible to stitch together a different, Votic-internal analogical explanation for the inessive. Kettunen in Vatjan kielen äännehistoria has done that already a century ago, starting from how we can find in Votic also a “strong-grade”  illative ending -sEE < *-sEn somewhat more widely than in the average Finnic variety. But generally I think a phonological explanation that takes care of multiple problems should be considered preferrable to unrelated analogical accounts, as long as no additional problems are introduced.
Moreover, if gradation of *st entered Votic as a relatively late analogy, we should perhaps expect it to fail to take root in environments where no analogical motivation for its extension was available. Such a case can indeed be found: the adverbial ending *-stik, found in Votic as simply -ssi(g). Contrary to Kettunen, early loss of final *-k cannot be blamed, since Eastern Votic is one of the few Finnic dialect groups where it in fact remains in some individual varieties: e.g. alassig ‘naked’ (and not ˣalazig).  Eastern Votic fails to apply gradation also to infinitives of s-stem verbs, e.g. pessäg ‘to wash’. Infinitives of resonant-stem verbs (e.g. tulla(g) ‘to come’, mennä(g) ‘to go’) would have provided a weak source of analogy though, explaining Western Votic pesä etc.
Even this last-mentioned analogy actually makes more sense if we assume it to have been earlier in relative chronology: from strong-grade *pestä(k) to weak-grade *pessä(k), rather than from strong-grade pessä(g) to weak-grade pesä. In the latter case I would definitely expect the pattern single consonant stem : geminate consonant infinitive to be more salient than the rather abstract grade difference. 
On the other hand, a “relatively late” date for the new look of *st-gradation could still be fairly early in absolute chronology. Evidence from Krevinian (an enclave dialect of Votic once spoken in Latvia, separated since the 15th century) suggests that the gradation pattern /ss/ : /s/ existed already by the late Middle Ages, and Kettunen even calls this “one of the oldest changes in Votic” (presumably meaning: one of the oldest uniquely Votic changes). But given that Proto-Finnic dates to around 0 CE, this still leaves plenty of time for the development of first the usual form of, and later a uniquely Votic type of *st-gradation.
There are a couple of what might look like chronological issues to this scenario. E.g. the introduction of /rt/ : /rr/ is, IIRC, perhaps as late as 17th century in parts of western Finland. Though I wonder if this should be taken as indication that, contrary to traditional Finnocentric default assumptions of all influence within Northern Finnic varieties having flowed from the west to the east, this gradation pattern is in part Karelian influence in western Finnish. (Perhaps even /lt/ : /ll/ and /nt/ : /nn/?!)
I also see it now and then assumed that Votic is the sole autochthonous language of Ingria, and that all varieties of Ingrian represent later intrusions from north of the Gulf of Finland (much like we know to be the case with Ingrian Finnish). However, at least the case of the Kukkuzi dialect is problematic. Traditionally analyzed as heavily Ingrianized Votic, it is also at least equally well analyzeable as a Voticized Northern Finnic variety. And a bit further west, the same problem arises with northeastern coastal Estonian as well: the dialect shows clear effects of Finnish contact, but also indications of original, more deep-reaching Northern Finnic affinity, e.g. the complete absense of õ, in whose place we find a perfectly etymological distribution of e versus o. (I wonder if anyone’s ever tried hunting for isoglosses connecting Kukkuzi and NECEs in particular.)
If Votic has continuously had Northern Finnic neighbors already since early on (early contacts are traditionally indeed assumed on lexical and morphological grounds, but this generally seems to have been considered to be a separate issue from contact with Ingrian in particular), again all the easier to assume that at one time, it too had the general “Central-Eastern Finnic” gradation pattern *st : *ss.
These days I even find myself wondering if all the other main dialects of Ingrian proper can really be derived from Karelia, even if we also count the Karelian Isthmus. Their differences strike me as sharper and starker than those between the individual dialects of Karelian. It seems clearly implausible to assume a single late migration, followed by rapid diversification within a minisculous geographic area. Assuming 3-4 wholly separate backmigrations would also be contrived, at least if we cannot find historical correspondences for them (like how the introduction of Ingrian Finnish has been connected to Ingria’s brief period as a part of the Kingdom of Sweden). Perhaps coastal Ingria in general was simply never Votic-speaking, and is in fact rather the original Eastern Finnic homeland…? A topic for another post, another time though.
 The even more widespread gradation pattern /ht/ : *hd must have a different explanation, though. There is, as far as I can tell, no evidence for a geminate **hh. I presume this pattern instead emerged very early, around the time middle Proto-Finnic *š had finished its trek backwards and settled as /h/, i.e. no longer an obstruent; and *d had in most Finnic varieties lenited to /ð/, but had not yet been lost. Around this time [ɦð] would have nicely paralleled clusters such as [lð] and also the likes of [ɦl], [rɦ]; especially if we assume that /ð/ did not phonologically hold the status of a dental fricative, but that of a dental approximant.
 As measured by the general patterning of Finnic isoglosses, not by raw geography. The latter method would leave Votic off at the southeastern fringe, and would probably put the center-of-mass of the innovation somewhere in southeastern Finland…
 South Estonian and Southern Ostrobothnian Finnish indicate the more archaic variant *-snA, but I would assume that this was established as a free or stylistic variant by Proto-Finnic, instead of being retained in an allomorphic distribution of some sort. South Estonian in particular appears to have been eager to extend consonant cluster assimilations from unstressed positions also to post-tonic positions (e.g. *koktu > kõtt ‘stomach’, *maksa > mass ‘liver’, *sakna > sann ‘sauna’), so it would be quite odd if here a “strong-grade” allomorph has instead been generalized from positions with secondary stress.
 I am not fully convinced that the Proto-Finnic alternation *s : *h, found in some suffixes and in *s-stem nominals, has anything to do with consonant gradation, given its total absense root-medially; not even in words like *vasikka (or *vasëikka?) ‘calf’, where there should not have been any possibility for the analogical reintroduction of /s/ from strong-grade forms. If the case of Fi. lähellä ~ läsnä ~ lästä ‘near’ would constitute an example doesn’t seem clear. For one, as cognates have only been found from Mari and Samoyedic, we do not know for certain if the root is to be reconstructed as *läšə- or *läsə-; and läsnä, lästä could potentially be explained as based on the inessive and elative (pre-PF *lähe-snä, *lähe-stä?) rather than continuing the archaic locative and ablative (pre-PF *läS-nä, *läS-tä). For two, these being adverbs, *s > *h in prosodically unstressed positions seems possible (as is probably the case for the 3rd person pronouns *hän, *hek).
 Similarly also e.g. Karelian alašti, not ˣalašši.
 A possibility that however remains is that the Eastern Votic forms might be rather analogical reintroductions, again on the basis of the geminate infinitive forms of n, l, r-stems. Still this seems to be “the wrong way around”, given that these forms retain final –g as a transparent trigger for gradation, while the weak-grade Western Votic forms do not.