Unlike Margate, just along the coast, Folkestone's creative plans for regeneration do not include the building of a swish gallery by a big-name architect. Instead, the town has taken a more subtle route. In 2008, backed by the Creative Foundation, whose chair is the local philanthropist Roger de Haan, it staged its first triennial, an event so joyful and clever its memory ha s outlasted, in my own case, that of pretty much all the art I've seen since. Scattered so as to make you feel that you alone had discovered each piece, the work was frequently beautiful, occasionally funny and always thought-provoking. Even better, it brought Folkestone's considerable charms – the town, after all, was once so grand it was the favoured holiday destination of Edward VII and his mistress, Alice Keppel – into sharp relief. Stumbling on all this art, so cunningly situated, the future and the past seemed suddenly to work together. I left feeling full of hope.
Three years on, and the Creative Foundation's wisdom is now obvious. Gallery or not, Folkestone is already well on its way to having something that Margate painfully lacks: a permanent collection. Several of the pieces that were commissioned for the 2008 triennial have remained in the town, most notably Tracey Emin's poignant Baby Things – a bonnet, booties and matinee jacket cast in bronze and then "abandoned" on railings and beneath park benches – and Mark Wallinger's Folk Stones, a collection of 19,240 numbered pebbles, each one representing a soldier killed on the first day of the Battle of the Somme, many of whom would have departed there from the town's harbour (Wallinger's piece has grown so beloved, it gets its own poppy wreath come Remembrance Sunday).
Meanwhile, the foundation's investment in existing real estate is slowly paying off. Fewer shops stand empty. On the harbour, a fabulous new restaurant has opened, its fish fresh off the boats each morning.
It pains me, then, to report that the second triennial is not quite so successful a proposition as the first. On the train, inspecting my blisters – there is a lot of walking to do: its brilliant and determined curator, Andrea Schlieker has commissioned 19 artists, some of whom have contributed more than one work – I tried to convince myself that disappointment was inevitable, given my ecstasy three years ago. But the truth is that this a more patchy affair. In 2008 artists were encouraged to use the town and its ghosts as their inspiration, something that more than justified the triennial's obsession with site specificity. In 2011 the feel is deliberately more outward-looking: its subtitle, after all, is A Million Miles From Home, a theme that nods both to Folkestone's geography – gazing out to France on a grey day, it can feel like the end of the world – and its status as a place where asylum seekers and other immigrants often end up.
The result, though, is a show that is sometimes off-puttingly preachy. The Israeli artist, Smadar Dreyfus, for instance, has recorded in their entirety seven lessons in Israeli schools, lessons that take in such loaded issues as citizenship and the Law of Return. Sitting in the pitch black of an abandoned office building listening to these lessons – a translation is provided on screen in the form of "word pictures" – is an object lesson in the bullying and self-indulgence involved in a certain kind of contemporary art. It's unendurable.
My advice? Avoid the film installations (I counted – yawn – four). Ditto the work that requires too lengthy explanation (in Folkestone's delightfully spooky Masonic Hall, a new and extremely winning venue in 2011, the artist Olivia Plender gamely tried to unpick for me her film installation Are Dreams Hallucinations During Sleep or Hallucinations Waking Dreams; alas, I am still none the wiser – all I can tell you is that it involves a local am dram group doing improvisations). No, head instead straight for the stuff that will hit you bang, smack in the solar plexus. Luckily, there is plenty of it.
The best work makes the most of Folkestone's beguiling topography. Begin your tour high on the Leas, almost as far west as HG Wells's house (designed by Voysey, it's now an old people's home), where you will find Cristina Iglesias's magical Towards the Sound of Wilderness. Iglesias has cut a path through undergrowth – a kind of secret passage – which leads to her "intervention", a terrific mirrored box-like structure whose walls, crafted to resemble thorns, call to mind The Sleeping Beauty. Step into it and, from a window, you will see a Martello tower now entirely covered in ivy. When the hundreds of birds that nest there sing, the experience is genuinely otherworldly.
From here, head east towards the Victorian water-powered Leas Lift. Riding the lift has always been a hairy experience – the descent is dramatic and always, somehow, unexpected – but now it's comical, too. As the mechanism begins to roll, so does Martin Creed's sound installation Work No. 1196, Piece for String Quartet and Elevator: a series of descending scales. Impossible not to smile.
Leave the lift, and you're on the site of the still-mourned Rotunda amusement park. Here stands AK Dolven's desolate and beautiful Out of Tune, a huge tenor bell suspended on wire between two beams. Pull the rope, and it will ring out, melancholy and ghostly. Dolven, who is Norwegian, speaks of having brought something – this old bell – back to life. But to me, it sounds more like a death knell, or a warning.
And perhaps it is. There is so much of Folkestone still to save. Close by is the old harbour railway station: hard to believe, standing among the rust and the weeds, that it was here that the Orient Express used to call. On the tracks is Paloma Varga Weisz's sublime Rug People, a group of men with Modigliani faces cast in bronze, standing on an oriental carpet. For all that is it so physically heavy, this sculpture, it seems to me, is the very embodiment of transience, a family's world reduced to the scant acreage of a patterned rug. I adored it, though it is Cornelia Parker's bronze, The Folkestone Mermaid, on Sunny Sands beach, that the townspeople will want to claim as their own. It's a delightful joke, of course, this nicking of Copenhagan's most famous landmark, but Parker has made a beautiful work in its own right. Strong, proud and human – no flipper for her, though her feet are draped with seaweed – this mermaid's jaw suggests the same patient indefatigability as that of the town she symbolises
Nearly there now. At the top of the Bayle, hanging in the nave of St Eanswythe's church, is Hew Locke's For Those in Peril on the Sea: a flotilla of votive offerings in the form of model boats in every shade and style you can imagine. It's a work that manages to be both impossibly cheery, and contemplative. Just below it, in the Old High Street, is Erzen Shkololli's Boutique Kosovo. Shkololli, who works in Pristina, has gathered together traditional Kosovan costumes, made by this country's craftswomen – except he has displayed them as if in some upscale minimalist store (think Marni). This is not the most nuanced commentary on globalisation that I've ever seen, but the garments are so fascinatingly exquisite – they seem almost to have the quality of religious relics – I'll forgive him. Besides, isn't this what we want for towns like Folkestone? Small (and possibly useful) shops rather than chain stores. Beauty rather than blandness. What you might call, feeling daring, a soul.