We’re pleased to note the release of AxiDraw software version 3.9 this week, with a couple of neat new features. One of them is that the “preview mode” button — which lets you simulate plotting to see how the results will come out — is now accessible no matter which function is selected. The other, … Continue reading Hidden line removal for AxiDraw
In 2005, Amazon introduced its Prime Free Shipping program. This yearly subscription program promised free two-day shipping on any purchase the subscriber made from Amazon. Five years later, 13% of Amazon’s 130 million active users are Prime members. More significantly, 20% of the subscribers who purchased products from Amazon in the last twelve months are Prime subscribers. These Prime subscribers purchase two to three times as much as non-Prime subscribers over the course of a year. This Performance innovation removes an impediment to purchasing on Amazon. In fact, it increases the odds greatly that online purchases will be made on Amazon rather than on a competitive site. This has been a blockbuster innovation for Amazon. The innovation holds a special appeal to the larger customers in the market. The Prime subscribers may also offer Amazon an entry into a business that it has longed to gain, for several years, subscription video rentals. It appears that Amazon will introduce a streaming video product for its Prime subscribers. This new product will not cost the Prime subscribers any more than their normal subscription. Netflix’s Watch Instantly service cost about $96 a year so Amazon may have a price advantage on Netflix. Of course, Convenience and Price are only important provided Amazon offers equivalent Function, that is, streaming video content. We don’t know about that yet. Still, Amazon has proven to be an innovative company who can find ways to build a business in non-traditional ways. It continues to grab market share in the retail business.
When lighting with small subjects, a speedlight is generally all you’ll ever need. Further, because of the scale and distances involved, even with a small flash you’ll be working at low power settings.
For these photos of some of the resident animals at the Howard County Conservancy, we used a single speedlight generally at one-eighth power, and let the environment of the impromptu studio to the rest of the heavy lifting for light modification.
Read more »
The final day of TIFF 2020 has come and gone and below are my final capsule reviews. I’ll post a full capsule roundup on Monday.
Fauna [Mexico/Canada, Nicolás Pereda, 3.5] Narratives nest within narratives when an actor visits his girlfriend’s family in a sleepy small town. Comic misunderstandings, naturalistic locations and twisting meta-story may remind seasoned festival-goers of the works of Hong Sang-soo, with Coronas instead of soju.
Preparations to Be Together For an Unknown Period of Time [Hungary, Lili Horvát, 4] Top neurologist questions the accuracy of her recollections when she moves back home from the US to Budapest for a romantic rendezvous, only to find that the object of her affections professes not to remember her. Quietly suspenseful drama of psychological uncertainty.
The Truffle Hunters [Italy, Michael Dweck & Gregory Kershaw, 4] An aging generation of Piedmontese truffle hunters carries on the search for the elusive delicacy, fearing the poison bait left for their beloved dogs by ruthless newcomers to the trade. A documentary balm for lovers of food and canines luxuriates in the presence of sumptuously photographed forest eccentrics and their very, very good dogs.
Bandar Band [Iran/Germany, Manijeh Hekmat, 3] A pregnant singer, her husband and their guitarist try to get their van through a floodstruck region to attend a contest gig in Tehran. Neorealist drama where the obstacles in the characters’ path are literal.
The Water Man [US, David Oyelowo, 3.5] Imaginative kid (Lonnie Chavis) heads into the Northwestern forest in search of a legendary immortal, thinking he holds the secret to curing his mom (Rosario Dawson) of leukemia. One of the more successful of a recent wave of films that put a somber sin on 80s kids adventure, thanks to a well-constructed script and Oyelowo’s sure control of tone.
Among the differences of this digital-only fest was that it removed the flexibility to choose between multiple screening dates. In a regular year I program the last days and work backward to end on some combination of stronger and/or lighter selections. Here programmers assigned a 24 hour window for each film. These last movies weren’t what I would have picked as closers in ordinary times. To compensate for this Valerie and I are running a day of fake TIFF programming to simulate the funner final Sunday we usually shoot for. They consist of one film that played at TIFF 2019 and three others from previously-appearing directors. Play along at home by streaming The Vast of Night, The Forest of Love*, Mr. & Mrs. Adelman, and Ace Attorney.
*Update: Turns out this one is ultra-disturbing and in no way fun or light. Going into something with mistaken tonal expectations—just like the real TIFF!
Capsule review boilerplate: Ratings are out of 5. I’ll be collecting these reviews in order of preference in a master post the Monday after the fest. Films shown on the festival circuit will appear in theaters, disc and/or streaming over the next year plus.
COVID has put the kibosh on much this year, but it can’t stop the capsule TIFF reviews. From the plague-ready, off-model edition of the Toronto International Film Festival, here’s my annual collection of mini-reviews.
The greatly cut-down slate included only slivers of the festival I’d program for myself in a regular year: four to five from international auteurs and a couple examples of global genre cinema. The missing items either are waiting in limbo as sales agents the world over hope that theatrical exhibition will return, or didn’t even get shot.
This list features more Canadian films and documentaries than I’d see at the fest (as opposed to catching them later.) Festivals tend toward the dour and downbeat but that was doubly true this time out. If we’re still trapped in our homes next year, I’ll likely be more vigilant about sorting through the slim pickings, supplementing our streaming experience with titles already available on other platforms.
That said, the overall hit rate was probably as strong as any other recent year. The average score on my numerical ratings would be higher, actually. It’s just that I saw the same festival everyone else did, starting with the film that garnered nearly universal acclaim, nabbed the People’s Choice Award, and will surely be part of the Oscars race—whatever the heck that will look like.
Films are listed in order of preference. Within categories that doesn’t mean much and entails a lot of apples-to-oranges comparisons. A festival near you, or not so near you but within your territory for geolocking purposes, may be virtually screening some of these soon.
Nomadland [US, Chloé Zhao, 5] When her town closes down in the wake of its gypsum mine’s closure, a self-reliant widow (Frances McDormand) moves into her van and joins the ranks of the nomad subculture, people who rove the US, taking whatever hard work they can get and living out of their vehicles. Rooted in social realist cinema, marked by a triad of transcendent qualities: poetic visual beauty, an indelible central performance and a deep love for the characters from the writer/director.
Another Round [Denmark, Thomas Vinterberg, 4.5] Burned out high school teacher (Mads Mikkelsen) embarks with three colleagues on an experiment to enhance their performance by maintaining a blood alcohol level of 0.5% throughout their days at work. Not only an original booze movie, but a big one, full of turns and ambiguities, and an utterly masterful performance from Mikkelsen.
City Hall [US, Frederick Wiseman, 4] The latest of Wiseman’s distinctive epic-length observational documentaries studies the quotidian, procedural and human moments of human life as seen through the processes of municipal government in Boston, as held together by the thoughtful charisma of Mayor Martin Walsh. Improbably absorbing as always, this institutional cross-section offers a beguiling vision of an oasis of good government in the USA.
True Mothers [Japan, Naomi Kawase, 4.5] Parents of a kindergartner react with dismay when a woman contacts them claiming to be his birth mother. Luminous, delicate drama of shifting perspectives.
Fireball: Visitors from Darker Worlds [US, Werner Herzog & Clive Oppenheimer] Documentary explores the science and mythology of meteor, from Chicxulub to ʻOumuamua. The intersection between scientific discovery and religious awe, central to all of Herzog’s beautiful and delightful nature docs, rises from subtext to text through the intercession of traditional elders, joyful researchers, and the Jesuit scholar of the Vatican’s heaven stone collection.
David Byrne’s American Utopia [US, Spike Lee, 4] Filmed version of the Broadway version of David Byrne’s recent tour features joyous choreography, simple but arresting stagecraft, and songs from his Talking Heads and solo eras. When you shoot a concert film featuring David Byrne, you have to bring it, and Lee does that ably, finding countlesss different ways to shoot within a proscenium.
The Father [UK, Florian Zeller, 4] Retired engineer (Anthony Hopkins) struggles to piece together the confusing reality of his living circumstances as his daughter (Olivia Colman) copes with his progressing dementia. Impeccably performed stage play adaptation puts the viewer inside the contradictory shifts of the protagonist’s subjective viewpoint.
Night of the Kings [Côte d'Ivoire/France , Philippe Lacôte, 4] When the red moon rises over MACA, the Ivory Coast’s toughest prison, its inmate boss appoints the new arrival as storyteller—a post that results in death if the tale ends before sundown. Prison drama with compelling narrative hook widens out to encompass ancient warfare, contemporary politics, and even a wizard duel.
Summer of 85 [France, Francois Ozon, 4] Love between two young men in a French beach town leads to a bizarre crime. Teen emotions run high in a sunlit melodrama of Eros and Thanatos.
Spring Blossom [France, Suzanne Lindon, 4] Bored with her classmates, an awkward 16 year old (played by the writer-director) pursues her attraction for a ruggedly handsome stage actor (Arnaud Valois.) Character drama sets aside the sexual aspect of this staple French cinema situation to focus on the emotion, periodically breaking from naturalism to have its characters express their feelings through dance.
Get the Hell Out [Taiwan, I-Fan Wang, 4] Taiwan’s notoriously pugilistic parliament tips into arterial spray when the effluent of a controversial chemical plant triggers a zombie epidemic. Zombie comedy features an eye-searing palette and an onslaught of optical overlays, and is paced like a quarter kilo of crushed Adderall.
Preparations to Be Together For an Unknown Period of Time [Hungary, Lili Horvát, 4] Top neurologist questions the accuracy of her recollections when she moves back home from the US to Budapest for a romantic rendezvous, only to find that the object of her affections professes not to remember her. Quietly suspenseful drama of psychological uncertainty.
Shiva Baby [US, Emma Seligman, 4] The ambient social pressures of a post-funeral gathering skyrocket for a directionless college student (Rachel Sennott) when attendees include not only the expected ex-girlfriend (Molly Gordon) but also the sex work client she’s caught feelings for. Knife-edge comedy of emotional suffocation uses a plucky suspense score for that extra frisson of social anxiety.
Under the Open Sky [Japan, Miwa Nishikawa, 4] Out of prison after a long sentence, an aging yakuza (Koji Yakusho) struggles with his volcanic temper as he attempts to go straight. Bittersweet drama anchored by a lead performance from Yakusho, a mainstay of contemporary Japanese cinema.
New Order [Mexico, Michel Franco, 4] A wedding thrown by a wealthy family during a growing insurrection suffers a murderous attack by protestors and the kidnapping of the bride. Wildly disturbing vision of political violence and degradation takes its time unreeling its allegorical purpose.
Limbo [UK, Ben Sharrock, 4] Syrian oud player grapples with guilt over family left behind as he cools his heels with other refugee claimants at a center in the bleak and isolated Outer Hebrides. Moments of deadpan humor and stark landscapes layer this exploration of displacement.
Violation [Canada, Madeleine Sims-Fewer & Dusty Mancinelli, 4] Woman (Madeleine Sims-Fewer) exacts meticulous revenge after her brother-in-law rapes her. Although this jarring, meditative drama includes gruesome imagery and horror-exploitation motifs, it’s closer in spirit to Michael Haneke than Dario Argento or Wes Craven.
Shadow in the Cloud [New Zealand, Roseanne Liang, 4] When an WWII RAF Flight Officer (Chloe Grace Moretz) boards a Samoa-bound cargo plane bearing a mysterious package, a monstrous gremlin on board is just one of the surprises. Enclosed space horror-action thriller tips an 80s-style hat to Carpenter and Cameron.
Beans [Canada, Tracey Deer, 4] As the 1990 Oka standoff envelops her Mohawk community, a shy tween achiever (Kiawentiio) decides to toughen up by ingratiating herself to the tough kids. Mixing the docudrama and coming-of-age structures offsets the inherent trickiness of both, but it wouldn’t work without an appealing and touching performance from its charismatic young lead.
Akilla’s Escape [Canada, Charles Officer, 4] Weed dealer hoping to leave the business (Saul Wiliiams) tries to recover his boss’ ripped-off cash and product without sacrificing a young gang member who reminds him of his younger self. Moody, laconic crime drama contextualized by the political history of Jamaican gangsterism.
Enemies of the State [US, Sonia Kennebeck] Documentary pulls apart a labyrinth of contradictory evidence around Matthew DeHart, an Indiana man who was framed for child pornography by the FBI as part of a Wikleaks espionage case, or created a story of secret files to shield himself either cooked up a Wikileaks-related espionage smokescreen to mask his sex crimes. Invites the viewer to join a filmmaking team as it goes ever deeper down a rabbit hole.
The Inconvenient Indian [Canada, Michelle Latimer, 4] Essay-format documentary examines the Indigenous struggle for sovereignty and cultural reclamation in North America, as hosted by novelist Thomas King and inspired by his nonfiction book of the same name. Makes its case through cinematic language, pushing the archival footage and talking heads format to the background.
Beginning [Georgia, Dea Kulumbegashvili, 4] Depressed wife of a pastor bears the brunt of a persecution campaign from a local man hostile to their minority Baptist faith. The camera acts as a pitiless eye in this harsh, austere drama of pervasive male oppression.
The Truffle Hunters [Italy, Michael Dweck & Gregory Kershaw, 4] An aging generation of Piedmontese truffle hunters carries on the search for the elusive delicacy, fearing the poison bait left for their beloved dogs by ruthless newcomers to the trade. A documentary balm for lovers of food and canines luxuriates in the presence of sumptuously photographed forest eccentrics and their very, very good dogs.
Lift Like a Girl [Egypt, Mayye Zayed, 4] From ages 13 to 18, under the tutelage of a volcanic, motormouth coach, with a rubble-strewn lot on a busy Alexandria street, weightlifter Zebiba trains to be a champion. Fly-on-the-wall documentary inhabits a hardscrabble community powered by loving verbal abuse.
The New Corporation: The Unfortunately Necessary Sequel [Canada, Joel Bakan & Jennifer Abbott, 4] Polemical documentary deploys narration, stock footage and talking heads (some appearing via lockdown video conference) to survey corporate capitalism and the struggle against it from Reaganomics to COVID and the George Floyd protests. Comprehensive primer for the prospective young progressives includes a call to continued electoral action.
40 Years a Prisoner [US, Tommy Oliver, 4] Documentary recounts the 1978 standoff between members of radical Black back-to-nature organization MOVE and Philadelphia police through the efforts of the son of two of the group members to secure their parole. A strong emotional hook greatly assists in telling a tenaciously complicated story.
Pieces of a Woman [US, Kornél Mundruczó, 3.5] Grief tears a couple (Vanessa KIrby, Shia LaBeouf) apart after the death of their baby in childbirth, abetted by the insistence of her domineering mother (Ellen Burstyn) that they pursue legal action against their midwife (Molly Parker.) Wrenching drama marked by deep performances and key long take scenes. An otherwise masterful script reaches for the conventional when it hits its climax.
Wildfire [UK/Ireland, Cathy Brady, 3.5] After going missing for a year, a bipolar woman (Nika McGuigan) drops in on her sister (Nora-Jane Noone), opening the wounds of shared tragedy. Raw, unsubtle family drama against the backdrop of Northern Irish politics as Brexit threatens a fragile peace.
Fauna [Mexico/Canada, Nicolás Pereda, 3.5] Narratives nest within narratives when an actor visits his girlfriend’s family in a sleepy small town. Comic misunderstandings, naturalistic locations and twisting meta-story may remind seasoned festival-goers of the works of Hong Sang-soo, with Coronas instead of soju.
The Water Man [US, David Oyelowo, 3.5] Imaginative kid (Lonnie Chavis) heads into the Northwestern forest in search of a legendary immortal, thinking he holds the secret to curing his mom (Rosario Dawson) of leukemia. One of the more successful of a recent wave of films that put a somber sin on 80s kids adventure, thanks to a well-constructed script and Oyelowo’s sure control of tone.
The Way I See It [US, Dawn Porter, 3.5] Documentary profile of Obama-era Official White House photographer traces his arc from work for the Reagan administration to anti-Trump social media firebrand. Whether American viewers consider this slickly fashioned film heartfelt or sentimental will depend on party registration. It’s certainly explicitly designed to fire up Ds to get out there to de-elect the current president.
Bandar Band [Iran/Germany, Manijeh Hekmat, 3] A pregnant singer, her husband and their guitarist try to get their van through a floodstruck region to attend a contest gig in Tehran. Neorealist drama where the obstacles in the characters’ path are literal.
Penguin Bloom [Australia, Glendyn Ivin, 3] A former surfer left paralyzed from the chest down by a freak accident reluctantly bonds with a magpie chick named Penguin, which one of her young sons has rescued. Sun-dappled animal-related family drama about the depression and anger that can accompany a life-changing injury.
Falling [US, Viggo Mortensen, 3] Pathologically forbearing airline pilot (Mortensen) attempts to find a new situation for his lifelong miserable prick of a father (Lance Henriksen) as his dementia worsens. With one character incapable of change and another not needing to change, almost all of the scenes repeat the same dynamic.
Gaza Mon Amor [Palestine/France, Tarzan & Arab Nasser, 3] Middle-aged fisherman discovers a Greek statue and courts a wary dress shop clerk. Deliberately paced dramedy of life under oppression.
Concrete Cowboy [US, Ricky Staub, 3] After yet another expulsion from school, a troubled teen (Caleb McLaughlin) gets dumped for the summer with his father (Idris Elba), who belongs to Philadelphia’s threatened culture of inner city horse owners. A rich social milieu is the star of the show in this affirming drama, which could do with a stronger drive to activate its protagonist.
I Care a Lot [UK, J Blakeson, 3] Corrupt legal guardian (Rosamund Pike) who slaps unsuspecting seniors into care facilities to bleed them dry triggers a cat-and-mouse game when her latest prey (Dianne Wiest) turns out to be the mother of a wealthy gangster (Peter Dinklage.) Engaging thriller— until it betrays the contract it has established with the audience.
Memory House [Brazil, João Paulo Miranda Maria, 1] Racist harassment from German co-workers drives dairy worker to vengeance. Blunts the political anger of its subject matter with enervating pacing.
For years TIFF has been intentionally or otherwise making it incrementally more difficult to do the event diehard style, as we have always done. Often it announces changes that blindside longtime loyalists—sometimes, as this year, after they’ve purchased their expensive memberships and ticket packages. This time around they surprised us by taking a tier of titles that any other year would be available through the package we purchased and moving them into a premium package previously reserved for the most publicized Gala films. For good measure, they threw in a couple of other medium-sized irritants.
To recreate something closer to our usual experience, we programmed another 25 titles already available on streaming platforms. Some actually played TIFF in the past; others were the types of movies that could have played the fest but didn’t.
I’m glad that we did, because the TIFF titles we were allowed to choose from included all of the duds of a normal year and none of the surprise masterpieces. Granted, it was a miracle that any films got made this year, and those that did tended toward the sorts of modest chamber pieces that could be produced under COVID protocol conditions.
This year crystallized a gradually growing realization we’ve been trying to suppress. So much has changed in the world of international cinema, from the festival’s position in their life cycle, to their subsequent availability, and even the style of the movies themselves, has completely changed since we started doing this in the mid 80s.
We have always gone to the fest for great films that we could otherwise never see, and started doing it in the VHS era. We don’t care about seeing things before anyone else does, or seeing the stars wave at us from the stage beforehand, or hearing audience members ask directors rambling questions afterwards. Even the virtues of a big screen experience are blunted by a dirty secret — a packed TIFF venue is not actually an ideal place to see a movie. Talkers and smartphone screens abound in every screening, and the bigger venues they convert into movie theaters for ten days are universally terrible.
You’d think that programming films from existing streaming platforms eliminates the other key part of fest-going, the surprise from out of nowhere. Except we got more of those with our alternate schedule than we did with the official titles this year.
In other words, after 36 years we are retiring from our vacation. Next year we’ll be doing a fully alternate replica of TIFF as we think of it from the past. The old rodeo is dead. Long live the new rodeo.
Here then is my final set of Toronto International Film Festival capsule reviews.
(Capsules for the 2021 Robin and Valerie International Film Festival will drop over time in Ken and Robin Consume Media.)
Murina (Croatia, Antoneta Alamat Kusijanović) Teen (Gracija Filipovic) chafes at the agitated authority of her command-barking father (Leon Lucev) as he hosts a rich, glamorous old friend (Cliff Curtis) to try to sell him on a resort proposal. Taut, superbly acted family drama set against the stunning yet slightly sinister beauty of the rocky Croatian coastline.
Saloum (Senegal, Jean Luc Herbulot) Three gunslingers—the mastermind, the hard case, and the magic user—take an unscheduled pit stop at an eccentric communal resort, which harbors horrible secrets of both the man-made and supernatural varieties. Gorgeously shot, tightly edited contemporary horror western with political resonance and cool monster design.
Compartment No. 6 (Finland, Juho Kuosmanen) Traveling alone on a trip she was supposed to take with her Muscovite professor girlfriend, a Finnish archaeology student finds herself sharing a compartment on the train to Murmansk with a loutish miner. Naturalistic light romantic drama of human connection overcoming barriers of class and personality.
OUT OF SYNC (Spain, Juanjo Giménez Peña) Isolated sound mixer (Marta Nieto) is unnerved to suffer a strange delay in her hearing, which becomes all the more inexplicable as it worsens. Realist weird tale makes smart use of cinema’s relationship between sight and sound.
Zalava (Iran, Arsalan Amiri) In pre-Revolutionary Iranian Kurdistan, a pig-headed police sergeant interferes with a djinn exorcism, sparking village hysteria. Tale of communal terror and its hazards generates suspense by skillfully modulating its pace.
Vengeance is Mine, All Others Pay Cash (Indonesia, Edwin) After meeting cute by beating the crap out of each other at a construction site, two lovers navigate the vicissitudes of fidelity, vengeance, and impotence. A martial arts flick that isn’t an action movie, but rather an allegorical romantic drama with elements of satire and magic realism to go with its bruising 70s style fights.
Hold Your Fire (US, Stefan Forbes) Documentary recreates the 1973 robbery-turned-hostage incident in which a group of young black Sunni men hoped to steal guns from a sporting goods shop to protect themselves from the Nation of Islam, in which the beginnings of hostage negotiation techniques were created and implemented on the fly. Archival footage and compelling retrospective interviews illuminate a complicated narrative with resonances into the present day.
A Banquet (UK, Ruth Paxton) After her husband’s death, a brittle woman (Sienna Guillory) struggles with her eldest daughter’s (Jessica Alexander) visionary transformation, which has taken away her need to eat. Slow burn realist cosmic horror filters eating disorders, emotional control and female rage.
Kicking Blood (Canada, Blaine Thurier) Vampire (Alanna Bale) connects with a detoxing alcoholic, prompting her to reconsider preying on humans. Frosty supernatural indie drama extends the vampire-as-addiction metaphor.
Yuni (Indonesia, Kamila Andini) High schooler with a yen for purple chafes at the narrow expectations her religious school, family and village have for her. Observational social drama enlivened by a vivid color palette.
The Daughter (Spain, Manuel Martín Cuenca) Teacher at a juvenile detention center helps a pregnant 14-year old escape so she can live secretly with him and his wife at their mountain home and give them the baby when it is born. Ultra-restrained domestic thriller could stand a notch or two less restraint.
Dug Dug (India, Ritwik Pareek) A local saint cult springs up when a motorbike keeps mysteriously returning to the site of its owner’s death. Gentle satire of faith and religious merchandising shows the sort of color and verve that raises hopes for a fresh wave of Indian art cinema.
Tug of War (Tanzania, Amil Shivji) A callow Marxist subversive falls for an Indian girl who has escaped her arranged marriage in British-controlled 1950s Zanzibar. Political romantic drama adopts the language of classic Hollywood glamor, albeit without the magnetic movie star performances the style depends on. Based on a classic Tanzanian novel.
Earwig (France, Lucile Hadžihalilović) In a creepy manor, an anxious loner (Paul Hilton) looks after a girl with teeth made of ice, at the behest of mysterious masters. The director’s first English language film pushes her dream narratives of childhood transformation into the far fringes of austerity.
You Are Not My Mother (Ireland, Kate Dolan) A withdrawn teen’s depressed mother briefly disappears, prefiguring the revelation of a supernatural family secret. Contemporary folk horror with stronger direction than script, with extensive foreshadowing genre fans will be well ahead of and an inactive menace that doesn’t do enough to propel the story.
Snakehead (US, Evan Jackson Leong) Smuggled immigrant (Shuya Chang) works off her debt by acting as the right hand to the matriarch (Jade Wu) of an NYC Chinatown crime family. Socially conscious gangland drama features the bane of longtime documentarians turning their hand to fiction: awkward, exposition-heavy scripting.
Arthur Rambo (France, Laurent Cantet) Rising literary star (Rabah Nait Oufella) plummets when the hate-filled tweets of his old alter ego resurface. Refined, uncinematic debate film presents thesis, antithesis, and credits.
Medusa (Brazil, Anita Rocha da Silveira) Member of AN ultra-right Christian school’s violent, pallid-masked theocratic girl gang goes undercover at a coma ward in search of a disfigured model. Overlong, unfocused political allegory references the horror genre, chiefly by adopting Dario Argento’s color palette.
After Blue (Dirty Paradise) (France, Bertrand Mandico) On a psychedelic alien world, a young outcast (Paula Luna) frees the statuesque, wish-granting death-dealer Kate Bush and she must accompany her hairdresser mother (Elina Löwensohn) on a quest to hunt her down. Invokes the spirits of Jodorowsky and Barbarella for a sleepwalk trudge through an arbitrary sequence of dream-logic events. Like its influences it is perhaps intended for a chemically altered audience.
La Soga 2 (US, Manny Perez) Dominican hitman (Perez) has gotten out and is living with a devoted new girlfriend, until a corrupt CIA officer pulls him back in. Scrappy microbudget crime flick is Dominican.
The Autumn 2022 issue of Agapé, the official journal of U.S. Grand Lodge O.T.O., is now available. This and all previous issues can be found here.
The Winter 2022 issue of Agapé, the official journal of U.S. Grand Lodge O.T.O., is now available. This and all previous issues can be found here.
The Spring 2023 issue of Agapé, the official journal of U.S. Grand Lodge O.T.O., is now available. This and all previous issues can be found here.
The Summer 2023 issue of Agapé, the official journal of U.S. Grand Lodge O.T.O., is now available. This and all previous issues can be found here.
I sat in my crumbling courthouse a couple of months ago, having edged past the permanently-stuck gate on the justices' car park, and made my way up the nearly-new lift to the assembly room. It is a handsome room, built in 1907 but has sadly not seen a lick of paint in the last decade-and-a-half and more.
Everywhere are signs of decay and neglect - but no matter. I understand the desperate need for the government to bring expenditure under control, even if that means denying resources to the public service that I have served unpaid these thirty years. There are still biscuits (amazingly) and most of the lights come on when you press a switch. There is some mysterious kit that we think might be for use in the new all-electronic courthouse. It still bears the protective film that we see on expensive audio visual stuff to protect it on its long journey from a Chinese sweatshop.
I have recently received an email from www.gov.uk/annual-tax-summary setting out the tax that I paid in the last fiscal year setting out the tax that I paid (direct tax only, so forget the taxes on consumption such as liquor duties and Council Tax (fifty quid a week on my modest Thames Valley bungalow).
Much more interesting is the breakdown of where it went, revealing how little our fellow citizens know of what is done with the country's collective cash.
Not that much goes on the justice system.
Last weekend I had the pleasure of being a panelist at the first annual Mississippi Book Festival at the State Capitol in Jackson, Mississippi!
For me, the weekend started with the plane flight out on Friday and a lovely reception that evening at the Eudora Welty House. Representatives of the Eudora Welty Foundation were on hand to provide tours and answer any and all questions about Jackson's favorite daughter. It was a great chance to talk to the organizers and volunteers, as well as other authors.
The next morning was breakfast at the Winter Archives Building, where the staff gave us a tour and showed us the forthcoming Museum of Civil Rights and Mississippi History Museum.
Then we were off to opening ceremonies, where the Jackson State University Marching Band performed on the Capitol steps, and then the panels! The Harper Lee Reconsidered panel, held in the old Supreme Court chamber, was lively and fascinating (and also covered by C-SPAN). I wasn't able to make it to the picture books panel due to the long line, but hear it went well, and I'd had the chance to talk with the presenters the night before :-).
My panel was the Young Readers panel, and featured moderator Margaret McMullan, and panelists Kimberly Willis Holt, Taylor Kitchings, Deborah Wiles, Carolyn Brown, and Cassie Beasley. Margaret did a great job as moderator and kept the conversation going and on track. :-).
Many thanks to all the organizers, volunteers, sponsors, and attendees for making the event such a success!
Altogether, it was a fantastic event, with standing-room-only crowds and a terrific venue! Here's a report on the festival from the Clarion-Ledger: Book Festival Attendance Outpaces Projections.
And here are some pics from out and about festival weekend:
| My duffel bag leaves the jetway in Houston |
| Art deco Greyhound Station, downtown Jackson |
| Kerry Madden, Susan Eaddy, Hester Bass, Chris Barton in the Eudora Welty House Garden |
| Deborah Wiles, Kerry Madden on the Eudora Welty House lawn |
| In front of the Eudora Welty House |
| MS State Capitol |
| Kerry Madden, Kimberly Willis Holt |
| W. Ralph Eubanks, Margaret McMullan |
| Jackson State University Marching Band |
| View from the Capitol steps |
| Capitol interior and dome |
| Dome in House of Representative Chamber |
| Dome of Senate Chamber |
| Mayflower Cafe |
| Kimberly, Taylor, Deborah, Margaret, Me, Cassie, Carolyn |
This past weekend, I had the pleasure of being one of the authors at the inaugural Lone Star Book Festival in Kingwood, Texas (just outside Houston)!
Here are some pics:
| Edward Carey, Emma Virjan, Jennifer Ziegler, Bethany Hegedus, Carmen Oliver, and me |
| Carmen Oliver presents BEARS MAKE THE BEST READING BUDDIES |
| Jennifer Ziegler and Jo Whittemore discuss encouraging reading |
| I present CHRONAL ENGINE and BORROWED TIME |
With all the advances in private space exploration, why do taxpayers still pay for space science missions?
How the inclusion of Katie Taylor v Amanda Serrano on the bill legitimises the carnival of Mike Tyson v Jake Paul in Texas
New projects:
Updated projects: 3b-bmfont, 3bgl-shader, 3bmd, 3d-math, 3d-spaces, 40ants-asdf-system, 40ants-slynk, access, acclimation, action-list, adhoc, adopt, adp, agnostic-lizard, alexandria, alexandria-plus, anatevka, anypool, april, arc-compat, architecture.builder-protocol, array-utils, arrow-macros, assoc-utils, async-process, atomics, auto-restart, aws-sdk-lisp, babel, bdef, bike, binary-structures, binding-arrows, birch, blackbird, bordeaux-threads, calm, carrier, caveman, ccldoc, cephes.cl, cepl, cerberus, cffi, cffi-object, cffi-ops, chanl, chunga, ci, ci-utils, ciao, cl-6502, cl-algebraic-data-type, cl-all, cl-ansi-term, cl-async, cl-atelier, cl-autowrap, cl-base32, cl-bmas, cl-bmp, cl-bnf, cl-brewer, cl-buchberger, cl-cmark, cl-collider, cl-colors2, cl-confidence, cl-containers, cl-cookie, cl-csv, cl-custom-hash-table, cl-cxx-jit, cl-data-structures, cl-dbi, cl-digraph, cl-dot, cl-enchant, cl-environments, cl-fast-ecs, cl-fbx, cl-fluent-logger, cl-form-types, cl-forms, cl-freetype2, cl-gamepad, cl-github-v3, cl-gltf, cl-gobject-introspection, cl-graph, cl-grip, cl-gserver, cl-hamcrest, cl-hash-util, cl-html-readme, cl-i18n, cl-info, cl-ini, cl-ipfs-api2, cl-kanren, cl-lib-helper, cl-liballegro, cl-liballegro-nuklear, cl-log, cl-markless, cl-marshal, cl-migratum, cl-mixed, cl-modio, cl-mount-info, cl-mpg123, cl-mssql, cl-mustache, cl-mysql, cl-neovim, cl-netpbm, cl-oju, cl-opengl, cl-opensearch-query-builder, cl-opus, cl-patterns, cl-plus-ssl-osx-fix, cl-ppcre, cl-project, cl-protobufs, cl-pslib, cl-pslib-barcode, cl-rashell, cl-readline, cl-sat.minisat, cl-sdl2-image, cl-sdl2-mixer, cl-sdl2-ttf, cl-sendgrid, cl-sentry-client, cl-skkserv, cl-smtp, cl-ssh-keys, cl-steamworks, cl-str, cl-svg, cl-telegram-bot, cl-threadpool, cl-tiled, cl-torrents, cl-tqdm, cl-transducers, cl-transit, cl-unicode, cl-unification, cl-unix-sockets, cl-utils, cl-vectors, cl-vorbis, cl-wavefront, cl-webdriver-client, cl-webkit, cl-webmachine, cl-who, clack, clack-pretend, clad, classimp, clast, clath, clavier, clazy, clerk, clgplot, climacs, clingon, clip, clj-con, clj-re, clobber, clog, clog-ace, clog-collection, clog-plotly, clog-terminal, clohost, closer-mop, clss, cluffer, clunit2, clx, cmd, codata-recommended-values, codex, coleslaw, collectors, colored, com-on, common-lisp-jupyter, commondoc-markdown, compiler-macro-notes, conduit-packages, consfigurator, contextl, croatoan, ctype, cytoscape-clj, damn-fast-priority-queue, dartscluuid, data-frame, data-lens, datafly, dbus, decompress, defenum, definer, definitions, deflate, defmain, deploy, depot, deptree, dexador, dissect, djula, dns-client, doc, docs-builder, dsm, dufy, easter-gauss, easy-audio, easy-macros, easy-routes, eclector, equals, erjoalgo-webutil, erudite, esrap, event-emitter, external-program, external-symbol-not-found, fare-csv, fare-scripts, fast-http, fast-websocket, file-attributes, file-notify, file-select, filesystem-utils, fiveam, fiveam-matchers, flexi-streams, float-features, flow, fn, fset, functional-trees, fuzzy-dates, gadgets, generic-cl, github-api-cl, glfw, glsl-toolkit, harmony, hashtrie, helambdap, http2, hunchentoot, imago, in-nomine, inferior-shell, introspect-environment, ironclad, jose, js, json-mop, jsonrpc, jzon, khazern, lack, lass, lemmy-api, letv, lichat-protocol, lichat-tcp-client, linear-programming, lisp-binary, lisp-chat, lisp-critic, lisp-pay, lisp-stat, lispcord, lla, local-time, log4cl-extras, logging, lru-cache, magicl, maiden, maidenhead, manifolds, math, mcclim, memory-regions, messagebox, method-combination-utilities, mgl-pax, misc-extensions, mito, mk-defsystem, mmap, mnas-package, mnas-string, moira, multiposter, mutility, mutils, named-closure, ndebug, neural-classifier, new-op, nibbles, nibbles-streams, ningle, nodgui, north, numerical-utilities, nytpu.lisp-utils, omglib, ook, open-location-code, openapi-generator, orizuru-orm, overlord, papyrus, parachute, parse-number, pathname-utils, petalisp, phos, picl, plot, plump, plump-sexp, pngload, policy-cond, polymorphic-functions, postmodern, ppath, prometheus-gc, psychiq, purgatory, py4cl, py4cl2, py4cl2-cffi, qlot, qoi, query-fs, quick-patch, quickhull, quri, random-state, reblocks, reblocks-auth, reblocks-file-server, reblocks-lass, reblocks-navigation-widget, reblocks-parenscript, reblocks-prometheus, reblocks-typeahead, reblocks-ui, reblocks-websocket, rove, s-dot2, sandalphon.lambda-list, sb-fastcgi, sc-extensions, sel, select, serapeum, shasht, shop3, si-kanren, sketch, slime, slite, sly, snooze, spinneret, staple, static-vectors, statistics, stepster, stmx, stripe, swank-crew, swank-protocol, sxql, symath, system-locale, taglib, teddy, ten, testiere, tfeb-lisp-hax, tfm, tiny-routes, tooter, trivia, trivial-arguments, trivial-clipboard, trivial-file-size, trivial-gray-streams, trivial-main-thread, trivial-octet-streams, trivial-package-locks, trivial-package-manager, trivial-sanitize, trivial-shell, type-templates, typo, uax-15, uiop, usocket, vellum, vellum-binary, vellum-csv, vellum-postmodern, verbose, vernacular, vom, websocket-driver, winhttp, with-branching, with-contexts, woo, xhtmlambda, xml-emitter, yason, zippy, zpb-ttf.
Removed projects: abstract-arrays, ahungry-fleece, cl-cheshire-cat, cl-darksky, cl-epoch, cl-naive-store, convolution-kernel, dense-arrays, extensible-compound-types, extensible-optimizing-coerce, fast-generic-functions, flac-metadata, freebsd-ffi, listoflist, luckless, one-more-re-nightmare, postmodern-localtime, stumpwm-dynamic-float, stumpwm-sndioctl, unicly.
To get this update, use:
(ql:update-dist "quicklisp")
Sorry this update took so long. My goal is to resume monthly releases.
Enjoy!
Common Lisp has an important language feature called dynamic binding. It is
possible to rebind a dynamic variable somewhere on the call stack and downstream
functions will see that new value, and when the stack is unwound, the old value
is brought back.
While Common Lisp does not specify multi-threading, it seems to be a consensus among various implementations that dynamic bindings are thread-local, allowing for controlling the computing context in a safe way.
Before we start experiments, let's define a package to isolate our namespace:
(defpackage "EU.TURTLEWARE.BLOG/DLET"
(:local-nicknames ("MOP" #+closer-mop "C2MOP"
#+(and (not closer-mop) ecl) "MOP"
#+(and (not closer-mop) ccl) "CCL"
#+(and (not closer-mop) sbcl) "SB-MOP"))
(:use "CL"))
(in-package "EU.TURTLEWARE.BLOG/DLET")
Dynamic binding of variables is transparent to the programmer, because the
operator LET is used for both lexical and dynamic bindings. For example:
(defvar *dynamic-variable* 42)
(defun test ()
(let ((*dynamic-variable* 15)
(lexical-variable 12))
(lambda ()
(print (cons *dynamic-variable* lexical-variable)))))
(funcall (test))
;;; (42 . 12)
(let ((*dynamic-variable* 'xx))
(funcall (test)))
;;; (xx . 12)
Additionally the language specifies a special operator PROGV that gives the
programmer a control over the dynamic binding mechanism, by allowing passing the
dynamic variable by value instead of its name. Dynamic variables are represented
by symbols:
(progv (list '*dynamic-variable*) (list 'zz)
(funcall (test)))
;;; (zz . 12)
Nowadays it is common to encapsulate the state in the instance of a class.
Sometimes that state is dynamic. It would be nice if we could use dynamic
binding to control it. That said slots are not variables, and if there are many
objects of the same class with different states, then using dynamic variables
defined with DEFVAR is not feasible.
Consider the following classes which we want to be thread-safe:
(defgeneric call-with-ink (cont window ink))
(defclass window-1 ()
((ink :initform 'red :accessor ink)))
(defmethod call-with-ink (cont (win window-1) ink)
(let ((old-ink (ink win)))
(setf (ink win) ink)
(unwind-protect (funcall cont)
(setf (ink win) old-ink))))
(defclass window-2 ()
())
(defvar *ink* 'blue)
(defmethod ink ((window window-2)) *ink*)
(defmethod call-with-ink (cont (win window-2) ink)
(let ((*ink* ink))
(funcall cont)))
The first example is clearly not thread safe. If we access the WINDOW-1
instance from multiple threads, then they will overwrite a value of the slot
INK.
The second example is not good either, because when we have many instances of
WINDOW-2 then they share the binding. Nesting CALL-WITH-INK will overwrite
the binding of another window.
The solution is to use PROGV:
(defclass window-3 ()
((ink :initform (gensym))))
(defmethod initialize-instance :after ((win window-3) &key)
(setf (symbol-value (slot-value win 'ink)) 'red))
(defmethod call-with-ink (cont (win window-3) ink)
(progv (list (slot-value win 'ink)) (list ink)
(funcall cont)))
This way each instance has its own dynamic variable that may be rebound with a
designated operator CALL-WITH-INK. It is thread-safe and private. We may add
some syntactic sugar so it is more similar to let:
(defmacro dlet (bindings &body body)
(loop for (var val) in bindings
collect var into vars
collect val into vals
finally (return `(progv (list ,@vars) (list ,@vals)
,@body))))
(defmacro dset (&rest pairs)
`(setf ,@(loop for (var val) on pairs by #'cddr
collect `(symbol-value ,var)
collect val)))
(defmacro dref (variable)
`(symbol-value ,variable))
While meta-classes are not easily composable, it is worth noting that we can mold it better into the language by specifying that slot itself has a dynamic value. This way CLOS aficionados will have a new tool in their arsenal.
The approach we'll take is that a fresh symbol is stored as the value of each instance-allocated slot, and then accessors for the slot value will use these symbols as a dynamic variable. Here are low-level accessors:
;;; Accessing and binding symbols behind the slot. We don't use SLOT-VALUE,
;;; because it will return the _value_ of the dynamic variable, and not the
;;; variable itself.
(defun slot-dvar (object slotd)
(mop:standard-instance-access
object (mop:slot-definition-location slotd)))
(defun slot-dvar* (object slot-name)
(let* ((class (class-of object))
(slotd (find slot-name (mop:class-slots class)
:key #'mop:slot-definition-name)))
(slot-dvar object slotd)))
(defmacro slot-dlet (bindings &body body)
`(dlet ,(loop for ((object slot-name) val) in bindings
collect `((slot-dvar* ,object ,slot-name) ,val))
,@body))
Now we'll define the meta-class. We need that to specialize functions responsible
for processing slot definitions and the instance allocation. Notice, that we
make use of a kludge to communicate between COMPUTE-EFFECTIVE-SLOT-DEFINITION
and EFFECTIVE-SLOT-DEFINITION-CLASS – this is because the latter has no
access to the direct slot definitions.
;;; The metaclass CLASS-WITH-DYNAMIC-SLOTS specifies alternative effective slot
;;; definitions for slots with an initarg :dynamic.
(defclass class-with-dynamic-slots (standard-class) ())
;;; Class with dynamic slots may be subclasses of the standard class.
(defmethod mop:validate-superclass ((class class-with-dynamic-slots)
(super standard-class))
t)
;;; When allocating the instance we initialize all slots to a fresh symbol that
;;; represents the dynamic variable.
(defmethod allocate-instance ((class class-with-dynamic-slots) &rest initargs)
(declare (ignore initargs))
(let ((object (call-next-method)))
(loop for slotd in (mop:class-slots class)
when (typep slotd 'dynamic-effective-slot) do
(setf (mop:standard-instance-access
object
(mop:slot-definition-location slotd))
(gensym (string (mop:slot-definition-name slotd)))))
object))
;;; To improve potential composability of CLASS-WITH-DYNAMIC-SLOTS with other
;;; metaclasses we treat specially only slots that has :DYNAMIC in initargs,
;;; otherwise we call the next method.
(defmethod mop:direct-slot-definition-class
((class class-with-dynamic-slots) &rest initargs)
(loop for (key val) on initargs by #'cddr
when (eq key :dynamic)
do (return-from mop:direct-slot-definition-class
(find-class 'dynamic-direct-slot)))
(call-next-method))
;;; The metaobject protocol did not specify an elegant way to communicate
;;; between the direct slot definition and the effective slot definition.
;;; Luckily we have dynamic bindings! :-)
(defvar *kludge/mop-deficiency/dynamic-slot-p* nil)
(defmethod mop:compute-effective-slot-definition
((class class-with-dynamic-slots)
name
direct-slotds)
(if (typep (first direct-slotds) 'dynamic-direct-slot)
(let* ((*kludge/mop-deficiency/dynamic-slot-p* t))
(call-next-method))
(call-next-method)))
(defmethod mop:effective-slot-definition-class
((class class-with-dynamic-slots) &rest initargs)
(declare (ignore initargs))
(if *kludge/mop-deficiency/dynamic-slot-p*
(find-class 'dynamic-effective-slot)
(call-next-method)))
Finally we define a direct and an effective slot classes, and specialize slot accessors that are invoked by the instance accessors.
;;; There is a considerable boilerplate involving customizing slots.
;;;
;;; - direct slot definition: local to a single defclass form
;;;
;;; - effective slot definition: combination of all direct slots with the same
;;; name in the class and its superclasses
;;;
(defclass dynamic-direct-slot (mop:standard-direct-slot-definition)
((dynamic :initform nil :initarg :dynamic :reader dynamic-slot-p)))
;;; DYNAMIC-EFFECTIVE-SLOT is implemented to return as slot-value values of the
;;; dynamic variable that is stored with the instance.
;;;
;;; It would be nice if we could specify :ALLOCATION :DYNAMIC for the slot, but
;;; then STANDARD-INSTANCE-ACCESS would go belly up. We could make a clever
;;; workaround, but who cares?
(defclass dynamic-effective-slot (mop:standard-effective-slot-definition)
())
(defmethod mop:slot-value-using-class
((class class-with-dynamic-slots)
object
(slotd dynamic-effective-slot))
(dref (slot-dvar object slotd)))
(defmethod (setf mop:slot-value-using-class)
(new-value
(class class-with-dynamic-slots)
object
(slotd dynamic-effective-slot))
(dset (slot-dvar object slotd) new-value))
(defmethod mop:slot-boundp-using-class
((class class-with-dynamic-slots)
object
(slotd dynamic-effective-slot))
(boundp (slot-dvar object slotd)))
(defmethod mop:slot-makunbound-using-class
((class class-with-dynamic-slots)
object
(slotd dynamic-effective-slot))
(makunbound (slot-dvar object slotd)))
With this, we can finally define a class with slots that have dynamic values. What's more, we may bind them like dynamic variables.
;;; Let there be light.
(defclass window-4 ()
((ink :initform 'red :dynamic t :accessor ink)
(normal :initform 'normal :accessor normal))
(:metaclass class-with-dynamic-slots))
(let ((object (make-instance 'window-4)))
(slot-dlet (((object 'ink) 15))
(print (ink object)))
(print (ink object)))
ContextL provides a similar solution with dynamic slots, although it provides much more, like layered classes. This example is much more self-contained.
Lately I'm working on the repaint queue for McCLIM. While doing so I've decided to make stream operations thread-safe, so it is possible to draw on the stream and write to it from arbitrary thread asynchronously. The access to the output record history needs to be clearly locked, so that may be solved by the mutex. Graphics state is another story, consider the following functions running from separate threads:
(defun team-red ()
(with-drawing-options (stream :ink +dark-red+)
(loop for i from 0 below 50000 do
(write-string (format nil "XXX: ~5d~%" i) stream))))
(defun team-blue ()
(with-drawing-options (stream :ink +dark-blue+)
(loop for i from 0 below 50000 do
(write-string (format nil "YYY: ~5d~%" i) stream))))
(defun team-pink ()
(with-drawing-options (stream :ink +deep-pink+)
(loop for i from 0 below 25000 do
(case (random 2)
(0 (draw-rectangle* stream 200 (* i 100) 250 (+ (* i 100) 50)))
(1 (draw-circle* stream 225 (+ (* i 100) 25) 25))))))
(defun gonow (stream)
(window-clear stream)
(time (let ((a (clim-sys:make-process #'team-red))
(b (clim-sys:make-process #'team-blue))
(c (clim-sys:make-process #'team-grue)))
(bt:join-thread a)
(bt:join-thread b)
(bt:join-thread c)
(format stream "done!~%"))) )
Operations like WRITE-STRING and DRAW-RECTANGLE can be implemented by
holding a lock over the shared resource without much disruption. The drawing
color on the other hand is set outside of the loop, so if we had locked the
graphics state with a lock, then these functions would be serialized despite
being called from different processes. The solution to this problem is to make
graphics context a dynamic slot that is accessed with WITH-DRAWING-OPTIONS.
I hope that I've convinced you that dynamic variables are cool (I'm sure that majority of readers here are already convinced), and that dynamic slots are even cooler :-). Watch forward to the upcoming McCLIM release!
If you like technical writeups like this, please consider supporting me on Patreon.
In the last post I've described a technique to use dynamic variables by value
instead of the name by utilizing the operator PROGV. Apparently it works fine
on all Common Lisp implementations I've tried except from SBCL, where the
number of thread local variables is by default limited to something below 4000.
To add salt to the injury, these variables are not garbage collected.
Try the following code to crash into LDB:
(defun foo ()
(loop for i from 0 below 4096 do
(when (zerop (mod i 100))
(print i))
(progv (list (gensym)) (list 42)
(values))))
(foo)
This renders our new technique not very practical given SBCL popularity. We
need to either abandon the idea or come up with a workaround.
Luckily for us we've already introduced a layer of indirection. Operators to
access dynamic variables are called DLET, DSET and DREF. This means, that
it is enough to provide a kludge implementation for SBCL with minimal changes
to the remaining code.
The old code works the same as previously except that instead of SYMBOL-VALUE
we use the accessor DYNAMIC-VARIABLE-VALUE, and the old call to PROGV is now
DYNAMIC-VARIABLE-PROGV. Moreover DYNAMIC-EFFECTIVE-SLOT used functions
BOUNDP and MAKUNBOUND, so we replace these with DYNAMIC-VARIABLE-BOUND-P
and DYNAMIC-VARIABLE-MAKUNBOUND. To abstract away things further we also
introduce the constructor MAKE-DYNAMIC-VARIABLE
(defpackage "EU.TURTLEWARE.BLOG/DLET"
(:local-nicknames ("MOP" #+closer-mop "C2MOP"
#+(and (not closer-mop) ecl) "MOP"
#+(and (not closer-mop) ccl) "CCL"
#+(and (not closer-mop) sbcl) "SB-MOP"))
(:use "CL"))
(in-package "EU.TURTLEWARE.BLOG/DLET")
(eval-when (:compile-toplevel :execute :load-toplevel)
(unless (member :bordeaux-threads *features*)
(error "Please load BORDEAUX-THREADS."))
(when (member :sbcl *features*)
(unless (member :fake-progv-kludge *features*)
(format t "~&;; Using FAKE-PROGV-KLUDGE for SBCL.~%")
(push :fake-progv-kludge *features*))))
(defmacro dlet (bindings &body body)
(flet ((pred (binding)
(and (listp binding) (= 2 (length binding)))))
(unless (every #'pred bindings)
(error "DLET: bindings must be lists of two values.~%~
Invalid bindings:~%~{ ~s~%~}" (remove-if #'pred bindings))))
(loop for (var val) in bindings
collect var into vars
collect val into vals
finally (return `(dynamic-variable-progv (list ,@vars) (list ,@vals)
,@body))))
(defmacro dset (&rest pairs)
`(setf ,@(loop for (var val) on pairs by #'cddr
collect `(dref ,var)
collect val)))
(defmacro dref (variable)
`(dynamic-variable-value ,variable))
;;; ...
(defmethod mop:slot-boundp-using-class
((class standard-class)
object
(slotd dynamic-effective-slot))
(dynamic-variable-bound-p (slot-dvar object slotd)))
(defmethod mop:slot-makunbound-using-class
((class standard-class)
object
(slotd dynamic-effective-slot))
(dynamic-variable-makunbound (slot-dvar object slotd)))
With these in place we can change the portable implementation to conform.
#-fake-progv-kludge
(progn
(defun make-dynamic-variable ()
(gensym))
(defun dynamic-variable-value (variable)
(symbol-value variable))
(defun (setf dynamic-variable-value) (value variable)
(setf (symbol-value variable) value))
(defun dynamic-variable-bound-p (variable)
(boundp variable))
(defun dynamic-variable-makunbound (variable)
(makunbound variable))
(defmacro dynamic-variable-progv (vars vals &body body)
`(progv ,vars ,vals ,@body)))
The implementation for SBCL will mediate access to the dynamic variable value with a synchronized hash table with weak keys. The current process is the key of the hash table and the list of bindings is the value of the hash table. For compatibility between implementations the top level value of the symbol will be shared.
The variable +FAKE-UNBOUND+ is the marker that signifies, that the variable
has no value. When the list of bindings is EQ to +CELL-UNBOUND+, then it
means that we should use the global value. We add new bindings by pushing to it.
#+fake-progv-kludge
(progn
(defvar +fake-unbound+ 'unbound)
(defvar +cell-unbound+ '(no-binding))
(defclass dynamic-variable ()
((tls-table
:initform (make-hash-table :synchronized t :weakness :key)
:reader dynamic-variable-tls-table)
(top-value
:initform +fake-unbound+
:accessor dynamic-variable-top-value)))
(defun make-dynamic-variable ()
(make-instance 'dynamic-variable))
(defun dynamic-variable-bindings (dvar)
(let ((process (bt:current-thread))
(tls-table (dynamic-variable-tls-table dvar)))
(gethash process tls-table +cell-unbound+)))
(defun (setf dynamic-variable-bindings) (value dvar)
(let ((process (bt:current-thread))
(tls-table (dynamic-variable-tls-table dvar)))
(setf (gethash process tls-table +cell-unbound+) value))))
We define two readers for the variable value - one that simply reads the value,
and the other that signals an error if the variable is unbound. Writer for its
value either replaces the current binding, or if the value cell is unbound, then
we modify the top-level symbol value. We use the value +FAKE-UNBOUND+ to check
whether the variable is bound and to make it unbound.
#+fake-progv-kludge
(progn
(defun %dynamic-variable-value (dvar)
(let ((tls-binds (dynamic-variable-bindings dvar)))
(if (eq tls-binds +cell-unbound+)
(dynamic-variable-top-value dvar)
(car tls-binds))))
(defun dynamic-variable-value (dvar)
(let ((tls-value (%dynamic-variable-value dvar)))
(when (eq tls-value +fake-unbound+)
(error 'unbound-variable :name "(unnamed)"))
tls-value))
(defun (setf dynamic-variable-value) (value dvar)
(let ((tls-binds (dynamic-variable-bindings dvar)))
(if (eq tls-binds +cell-unbound+)
(setf (dynamic-variable-top-value dvar) value)
(setf (car tls-binds) value))))
(defun dynamic-variable-bound-p (dvar)
(not (eq +fake-unbound+ (%dynamic-variable-value dvar))))
(defun dynamic-variable-makunbound (dvar)
(setf (dynamic-variable-value dvar) +fake-unbound+)))
Finally we define the operator to dynamically bind variables that behaves
similar to PROGV. Note that we PUSH and POP from the thread-local hash
table DYNAMIC-VARIABLE-BINDINGS, so no synchronization is necessary.
#+fake-progv-kludge
(defmacro dynamic-variable-progv (vars vals &body body)
(let ((svars (gensym))
(svals (gensym))
(var (gensym))
(val (gensym)))
`(let ((,svars ,vars))
(loop for ,svals = ,vals then (rest ,svals)
for ,var in ,svars
for ,val = (if ,svals (car ,svals) +fake-unbound+)
do (push ,val (dynamic-variable-bindings ,var)))
(unwind-protect (progn ,@body)
(loop for ,var in ,svars
do (pop (dynamic-variable-bindings ,var)))))))
But of course, we are going to also write a test framework. It's short, I
promise. As a bonus point the API is compatibile with fiveam, so it is
possible to drop tests as is in the appropriate test suite.
(defvar *all-tests* '())
(defun run-tests ()
(dolist (test (reverse *all-tests*))
(format *debug-io* "Test ~a... " test)
(handler-case (funcall test)
(serious-condition (c)
(format *debug-io* "Failed: ~a~%" c))
(:no-error (&rest args)
(declare (ignore args))
(format *debug-io* "Passed.~%")))))
(defmacro test (name &body body)
`(progn
(pushnew ',name *all-tests*)
(defun ,name () ,@body)))
(defmacro is (form)
`(assert ,form))
(defmacro pass ())
(defmacro signals (condition form)
`(is (block nil
(handler-case ,form
(,condition () (return t)))
nil)))
(defmacro finishes (form)
`(is (handler-case ,form
(serious-condition (c)
(declare (ignore c))
nil)
(:no-error (&rest args)
(declare (ignore args))
t))))
Now let's get to tests. First we'll test our metaclass:
(defclass dynamic-let.test-class ()
((slot1 :initarg :slot1 :dynamic nil :accessor slot1)
(slot2 :initarg :slot2 :dynamic t :accessor slot2)
(slot3 :initarg :slot3 :accessor slot3))
(:metaclass class-with-dynamic-slots))
(defparameter *dynamic-let.test-instance-1*
(make-instance 'dynamic-let.test-class
:slot1 :a :slot2 :b :slot3 :c))
(defparameter *dynamic-let.test-instance-2*
(make-instance 'dynamic-let.test-class
:slot1 :x :slot2 :y :slot3 :z))
(test dynamic-let.1
(let ((o1 *dynamic-let.test-instance-1*)
(o2 *dynamic-let.test-instance-2*))
(with-slots (slot1 slot2 slot3) o1
(is (eq :a slot1))
(is (eq :b slot2))
(is (eq :c slot3)))
(with-slots (slot1 slot2 slot3) o2
(is (eq :x slot1))
(is (eq :y slot2))
(is (eq :z slot3)))))
(test dynamic-let.2
(let ((o1 *dynamic-let.test-instance-1*)
(o2 *dynamic-let.test-instance-2*))
(signals error (slot-dlet (((o1 'slot1) 1)) nil))
(slot-dlet (((o1 'slot2) :k))
(is (eq :k (slot-value o1 'slot2)))
(is (eq :y (slot-value o2 'slot2))))))
(test dynamic-let.3
(let ((o1 *dynamic-let.test-instance-1*)
(exit nil)
(fail nil))
(flet ((make-runner (values)
(lambda ()
(slot-dlet (((o1 'slot2) :start))
(let ((value (slot2 o1)))
(unless (eq value :start)
(setf fail value)))
(loop until (eq exit t) do
(setf (slot2 o1) (elt values (random (length values))))
(let ((value (slot2 o1)))
(unless (member value values)
(setf fail value)
(setf exit t))))))))
(let ((r1 (bt:make-thread (make-runner '(:k1 :k2))))
(r2 (bt:make-thread (make-runner '(:k3 :k4))))
(r3 (bt:make-thread (make-runner '(:k5 :k6)))))
(sleep .1)
(setf exit t)
(map nil #'bt:join-thread (list r1 r2 r3))
(is (eq (slot2 o1) :b))
(is (null fail))))))
Then let's test the dynamic variable itself:
(test dynamic-let.4
"Test basic dvar operators."
(let ((dvar (make-dynamic-variable)))
(is (eql 42 (dset dvar 42)))
(is (eql 42 (dref dvar)))
(ignore-errors
(dlet ((dvar :x))
(is (eql :x (dref dvar)))
(error "foo")))
(is (eql 42 (dref dvar)))))
(test dynamic-let.5
"Test bound-p operator."
(let ((dvar (make-dynamic-variable)))
(is (not (dynamic-variable-bound-p dvar)))
(dset dvar 15)
(is (dynamic-variable-bound-p dvar))
(dynamic-variable-makunbound dvar)
(is (not (dynamic-variable-bound-p dvar)))))
(test dynamic-let.6
"Test makunbound operator."
(let ((dvar (make-dynamic-variable)))
(dset dvar t)
(is (dynamic-variable-bound-p dvar))
(finishes (dynamic-variable-makunbound dvar))
(is (not (dynamic-variable-bound-p dvar)))))
(test dynamic-let.7
"Test locally bound-p operator."
(let ((dvar (make-dynamic-variable)))
(is (not (dynamic-variable-bound-p dvar)))
(dlet ((dvar 15))
(is (dynamic-variable-bound-p dvar)))
(is (not (dynamic-variable-bound-p dvar)))))
(test dynamic-let.8
"Test locally unbound-p operator."
(let ((dvar (make-dynamic-variable)))
(dset dvar t)
(is (dynamic-variable-bound-p dvar))
(dlet ((dvar nil))
(is (dynamic-variable-bound-p dvar))
(finishes (dynamic-variable-makunbound dvar))
(is (not (dynamic-variable-bound-p dvar))))
(is (dynamic-variable-bound-p dvar))))
(test dynamic-let.9
"Stress test the implementation (see :FAKE-PROGV-KLUDGE)."
(finishes ; at the same time
(let ((dvars (loop repeat 4096 collect (make-dynamic-variable))))
;; ensure tls variable
(loop for v in dvars do
(dlet ((v 1))))
(loop for i from 0 below 4096
for r = (random 4096)
for v1 in dvars
for v2 = (elt dvars r) do
(when (zerop (mod i 64))
(pass))
(dlet ((v1 42)
(v2 43))
(values))))))
(test dynamic-let.0
"Stress test the implementation (see :FAKE-PROGV-KLUDGE)."
(finishes ; can be gc-ed
(loop for i from 0 below 4096 do
(when (zerop (mod i 64))
(pass))
(dlet (((make-dynamic-variable) 42))
(values)))))
All that is left is to test both dynamic variable implementations:
BLOG/DLET> (lisp-implementation-type)
"ECL"
BLOG/DLET> (run-tests)
Test DYNAMIC-LET.1... Passed.
Test DYNAMIC-LET.2... Passed.
Test DYNAMIC-LET.3... Passed.
Test DYNAMIC-LET.4... Passed.
Test DYNAMIC-LET.5... Passed.
Test DYNAMIC-LET.6... Passed.
Test DYNAMIC-LET.7... Passed.
Test DYNAMIC-LET.8... Passed.
Test DYNAMIC-LET.9... Passed.
Test DYNAMIC-LET.0... Passed.
NIL
And with the kludge:
BLOG/DLET> (lisp-implementation-type)
"SBCL"
BLOG/DLET> (run-tests)
Test DYNAMIC-LET.1... Passed.
Test DYNAMIC-LET.2... Passed.
Test DYNAMIC-LET.3... Passed.
Test DYNAMIC-LET.4... Passed.
Test DYNAMIC-LET.5... Passed.
Test DYNAMIC-LET.6... Passed.
Test DYNAMIC-LET.7... Passed.
Test DYNAMIC-LET.8... Passed.
Test DYNAMIC-LET.9... Passed.
Test DYNAMIC-LET.0... Passed.
NIL
In this post we've made our implementation to work on SBCL even when there are more than a few thousand dynamic variables. We've also added a simple test suite that checks the basic behavior.
As it often happens, after achieving some goal we get greedy and achieve more. That's the case here as well. In the next (and the last) post in this series I'll explore the idea of adding truly thread-local variables without a shared global value. This will be useful for lazily creating context on threads that are outside of our control. We'll also generalize the implementation so it is possible to subclass and implement ones own flavor of a dynamic variable.
In the previous two posts I've presented an implementation of first-class
dynamic variables using PROGV and a surrogate implementation for SBCL.
Now we will double down on this idea and make the protocol extensible. Finally we'll implement a specialized version of dynamic variables where even the top level value of the variable is thread-local.
Previously we've defined operators as either macros or functions. Different implementations were protected by the feature flag and symbols collided. Now we will introduce the protocol composed of a common superclass and functions that are specialized by particular implementations.
Most notably we will introduce a new operator CALL-WITH-DYNAMIC-VARIABLE that
is responsible for establishing a single binding. Thanks to that it will be
possible to mix dynamic variables of different types within a single DLET
statement.
(defclass dynamic-variable () ())
(defgeneric dynamic-variable-bindings (dvar))
(defgeneric dynamic-variable-value (dvar))
(defgeneric (setf dynamic-variable-value) (value dvar))
(defgeneric dynamic-variable-bound-p (dvar))
(defgeneric dynamic-variable-makunbound (dvar))
(defgeneric call-with-dynamic-variable (cont dvar &optional value))
Moreover we'll define a constructor that is specializable by a key. This design will allow us to refer to the dynamic variable class by using a shorter name. We will also define the standard class to be used and an matching constructor.
(defparameter *default-dynamic-variable-class*
#-fake-progv-kludge 'standard-dynamic-variable
#+fake-progv-kludge 'surrogate-dynamic-variable)
(defgeneric make-dynamic-variable-using-key (key &rest initargs)
(:method (class &rest initargs)
(apply #'make-instance class initargs))
(:method ((class (eql t)) &rest initargs)
(apply #'make-instance *default-dynamic-variable-class* initargs))
(:method ((class null) &rest initargs)
(declare (ignore class initargs))
(error "Making a dynamic variable that is not, huh?")))
(defun make-dynamic-variable (&rest initargs)
(apply #'make-dynamic-variable-using-key t initargs))
Control operators are the same as previously, that is a set of four macros that
consume the protocol specified above. Note that DYNAMIC-VARIABLE-PROGV expands
to a recursive call where each binding is processed separately.
(defmacro dlet (bindings &body body)
(flet ((pred (binding)
(and (listp binding) (= 2 (length binding)))))
(unless (every #'pred bindings)
(error "DLET: bindings must be lists of two values.~%~
Invalid bindings:~%~{ ~s~%~}" (remove-if #'pred bindings))))
(loop for (var val) in bindings
collect var into vars
collect val into vals
finally (return `(dynamic-variable-progv (list ,@vars) (list ,@vals)
,@body))))
(defmacro dset (&rest pairs)
`(setf ,@(loop for (var val) on pairs by #'cddr
collect `(dref ,var)
collect val)))
(defmacro dref (variable)
`(dynamic-variable-value ,variable))
(defun call-with-dynamic-variable-progv (cont vars vals)
(flet ((thunk ()
(if vals
(call-with-dynamic-variable cont (car vars) (car vals))
(call-with-dynamic-variable cont (car vars)))))
(if vars
(call-with-dynamic-variable-progv #'thunk (cdr vars) (cdr vals))
(funcall cont))))
(defmacro dynamic-variable-progv (vars vals &body body)
(let ((cont (gensym)))
`(flet ((,cont () ,@body))
(call-with-dynamic-variable-progv (function ,cont) ,vars ,vals))))
Previously we've used SBCL-specific options to define a synchronized hash table with weak keys. This won't do anymore, because we will need a similar object to implement the thread-local storage for top level values.
trivial-garbage is a portability layer that allows to define hash tables with
a specified weakness, but it does not provide an argument that would abstract
away synchronization. We will ensure thread-safety with locks instead.
(defclass tls-table ()
((table :initform (trivial-garbage:make-weak-hash-table
:test #'eq :weakness :key))
(lock :initform (bt:make-lock))))
(defun make-tls-table ()
(make-instance 'tls-table))
(defmacro with-tls-table ((var self) &body body)
(let ((obj (gensym)))
`(let* ((,obj ,self)
(,var (slot-value ,obj 'table)))
(bt:with-lock-held ((slot-value ,obj 'lock)) ,@body))))
STANDARD-DYNAMIC-VARIABLEPreviously in the default implementation we've represented dynamic variables
with a symbol. The new implementation is similar except that the symbol is read
from a STANDARD-OBJECT that represents the variable. This also enables us to
specialize the function CALL-WITH-DYNAMIC-VARIABLE:
(defclass standard-dynamic-variable (dynamic-variable)
((symbol :initform (gensym) :accessor dynamic-variable-bindings)))
(defmethod dynamic-variable-value ((dvar standard-dynamic-variable))
(symbol-value (dynamic-variable-bindings dvar)))
(defmethod (setf dynamic-variable-value) (value (dvar standard-dynamic-variable))
(setf (symbol-value (dynamic-variable-bindings dvar)) value))
(defmethod dynamic-variable-bound-p ((dvar standard-dynamic-variable))
(boundp (dynamic-variable-bindings dvar)))
(defmethod dynamic-variable-makunbound ((dvar standard-dynamic-variable))
(makunbound (dynamic-variable-bindings dvar)))
(defmethod call-with-dynamic-variable (cont (dvar standard-dynamic-variable)
&optional (val nil val-p))
(progv (list (dynamic-variable-bindings dvar)) (if val-p (list val) ())
(funcall cont)))
SURROGATE-DYNAMIC-VARIABLEThe implementation of the SURROGATE-DYNAMIC-VARIABLE is almost the same as
previously. The only difference is that we use the previously defined
indirection to safely work with hash tables. Also note, that we are not add the
feature condition - both classes is always created.
(defvar +fake-unbound+ 'unbound)
(defvar +cell-unbound+ '(no-binding))
(defclass surrogate-dynamic-variable (dynamic-variable)
((tls-table
:initform (make-tls-table)
:reader dynamic-variable-tls-table)
(top-value
:initform +fake-unbound+
:accessor dynamic-variable-top-value)))
(defmethod dynamic-variable-bindings ((dvar surrogate-dynamic-variable))
(let ((process (bt:current-thread)))
(with-tls-table (tls-table (dynamic-variable-tls-table dvar))
(gethash process tls-table +cell-unbound+))))
(defmethod (setf dynamic-variable-bindings) (value (dvar surrogate-dynamic-variable))
(let ((process (bt:current-thread)))
(with-tls-table (tls-table (dynamic-variable-tls-table dvar))
(setf (gethash process tls-table) value))))
(defun %dynamic-variable-value (dvar)
(let ((tls-binds (dynamic-variable-bindings dvar)))
(if (eq tls-binds +cell-unbound+)
(dynamic-variable-top-value dvar)
(car tls-binds))))
(defmethod dynamic-variable-value ((dvar surrogate-dynamic-variable))
(let ((tls-value (%dynamic-variable-value dvar)))
(when (eq tls-value +fake-unbound+)
(error 'unbound-variable :name "(unnamed)"))
tls-value))
(defmethod (setf dynamic-variable-value) (value (dvar surrogate-dynamic-variable))
(let ((tls-binds (dynamic-variable-bindings dvar)))
(if (eq tls-binds +cell-unbound+)
(setf (dynamic-variable-top-value dvar) value)
(setf (car tls-binds) value))))
(defmethod dynamic-variable-bound-p ((dvar surrogate-dynamic-variable))
(not (eq +fake-unbound+ (%dynamic-variable-value dvar))))
(defmethod dynamic-variable-makunbound ((dvar surrogate-dynamic-variable))
(setf (dynamic-variable-value dvar) +fake-unbound+))
;;; Apparently CCL likes to drop^Helide some writes and that corrupts bindings
;;; table. Let's ensure that the value is volatile.
#+ccl (defvar *ccl-ensure-volatile* nil)
(defmethod call-with-dynamic-variable (cont (dvar surrogate-dynamic-variable)
&optional (val +fake-unbound+))
(push val (dynamic-variable-bindings dvar))
(let (#+ccl (*ccl-ensure-volatile* (dynamic-variable-bindings dvar)))
(unwind-protect (funcall cont)
(pop (dynamic-variable-bindings dvar)))))
We've refactored the previous code to be extensible. Now we can use metaobjects from the previous post without change. We can also test both implementations in the same process interchangeably by customizing the default class parameter.
It is the time now to have some fun and extend dynamic variables into variables with top value not shared between different threads. This will enable ultimate thread safety. With our new protocol the implementation is trivial!
First we will define the protocol class. THREAD-LOCAL-VARIABLE is a variant of
a DYNAMIC-VARIABLE with thread-local top values.
We specify initialization arguments :INITVAL and :INITFUN that will be used
to assign the top value of a binding. The difference is that INITVAL specifies
a single value, while INITFUN can produce an unique object on each invocation.
INITARG takes a precedence over INTIFUN, and if neither is supplied, then a
variable is unbound.
We include the constructor that builds on MAKE-DYNAMIC-VARIABLE-USING-KEY, and
macros corresponding to DEFVAR and DEFPARAMETER. Note that they expand to
:INITFUN - this assures that the initialization form is re-evaluated for each
new thread where the variable is used.
(defclass thread-local-variable (dynamic-variable) ())
(defmethod initialize-instance :after
((self thread-local-variable) &key initfun initval)
(declare (ignore self initfun initval)))
(defparameter *default-thread-local-variable-class*
#-fake-progv-kludge 'standard-thread-local-variable
#+fake-progv-kludge 'surrogate-thread-local-variable)
(defun make-thread-local-variable (&rest initargs)
(apply #'make-dynamic-variable-using-key
*default-thread-local-variable-class* initargs))
(defmacro create-tls-variable (&optional (form nil fp) &rest initargs)
`(make-thread-local-variable
,@(when fp `(:initfun (lambda () ,form)))
,@initargs))
(defmacro define-tls-variable (name &rest initform-and-initargs)
`(defvar ,name (create-tls-variable ,@initform-and-initargs)))
(defmacro define-tls-parameter (name &rest initform-and-initargs)
`(defparameter ,name (create-tls-variable ,@initform-and-initargs)))
Perhaps it is a good time to introduce a new convention for tls variable names. I think that surrounding names with the minus sign is a nice idea, because it signifies, that it is something less than a global value. For example:
DYNAMIC-VARS> (define-tls-variable -context-
(progn
(print "Initializing context!")
(list :context)))
-CONTEXT-
DYNAMIC-VARS> -context-
#<a EU.TURTLEWARE.DYNAMIC-VARS::STANDARD-THREAD-LOCAL-VARIABLE 0x7f7636c08640>
DYNAMIC-VARS> (dref -context-)
"Initializing context!"
(:CONTEXT)
DYNAMIC-VARS> (dref -context-)
(:CONTEXT)
DYNAMIC-VARS> (dset -context- :the-new-value)
:THE-NEW-VALUE
DYNAMIC-VARS> (dref -context-)
:THE-NEW-VALUE
DYNAMIC-VARS> (bt:make-thread
(lambda ()
(print "Let's read it!")
(print (dref -context-))))
#<process "Anonymous thread" 0x7f7637a26cc0>
"Let's read it!"
"Initializing context!"
(:CONTEXT)
DYNAMIC-VARS> (dref -context-)
:THE-NEW-VALUE
You might have noticed the inconspicuous operator DYNAMIC-VARIABLE-BINDINGS
that is part of the protocol. It returns an opaque object that represents values
of the dynamic variable in the current context:
STANDARD-DYNAMIC-VARIABLE it is a symbolSURROGATE-DYNAMIC-VARIABLE it is a thread-local list of bindingsIn any case all other operators first take this object and then use it to read,
write or bind the value. The gist of the tls variables implementation is to
always return an object that is local to the thread. To store these objects we
will use the tls-table we've defined earlier.
(defclass thread-local-variable-mixin (dynamic-variable)
((tls-table
:initform (make-tls-table)
:reader dynamic-variable-tls-table)
(tls-initfun
:initarg :initfun
:initform nil
:accessor thread-local-variable-initfun)
(tls-initval
:initarg :initval
:initform +fake-unbound+
:accessor thread-local-variable-initval)))
For the class STANDARD-THREAD-LOCAL-VARIABLE we will simply return a
different symbol depending on the thread:
(defclass standard-thread-local-variable (thread-local-variable-mixin
thread-local-variable
standard-dynamic-variable)
())
(defmethod dynamic-variable-bindings ((tvar standard-thread-local-variable))
(flet ((make-new-tls-bindings ()
(let ((symbol (gensym))
(initval (thread-local-variable-initval tvar))
(initfun (thread-local-variable-initfun tvar)))
(cond
((not (eq +fake-unbound+ initval))
(setf (symbol-value symbol) initval))
((not (null initfun))
(setf (symbol-value symbol) (funcall initfun))))
symbol)))
(let ((key (bt:current-thread)))
(with-tls-table (tls-table (dynamic-variable-tls-table tvar))
(or (gethash key tls-table)
(setf (gethash key tls-table)
(make-new-tls-bindings)))))))
And for the class SURROGATE-THREAD-LOCAL-VARIABLE the only difference from the
SURROGATE-DYNAMIC-VARIABLE implementation is to cons a new list as the initial
value (even when it is unbound) to ensure it is not EQ to +CELL-UNBOUND+.
(defclass surrogate-thread-local-variable (thread-local-variable-mixin
thread-local-variable
surrogate-dynamic-variable)
())
(defmethod dynamic-variable-bindings ((tvar surrogate-thread-local-variable))
(flet ((make-new-tls-bindings ()
(let ((initval (thread-local-variable-initval tvar))
(initfun (thread-local-variable-initfun tvar)))
(cond
((not (eq +fake-unbound+ initval))
(list initval))
((not (null initfun))
(list (funcall initfun)))
(t
(list +fake-unbound+))))))
(let ((key (bt:current-thread)))
(with-tls-table (tls-table (dynamic-variable-tls-table tvar))
(or (gethash key tls-table)
(setf (gethash key tls-table)
(make-new-tls-bindings)))))))
That's all, now we have two implementations of thread-local variables.
Ramifications are similar as with "ordinary" dynamic variables - the standard
implementation is not advised for SBCL, because it will crash in LDB.
First we are going to allow to defined dynamic variable types with an
abbreviated names. This will enable us to specify in the slot definition that
type, for example (MY-SLOT :DYNAMIC :TLS :INITFORM 34)
;;; Examples how to add shorthand type names for the dynamic slots:
(defmethod make-dynamic-variable-using-key ((key (eql :tls)) &rest initargs)
(apply #'make-dynamic-variable-using-key
*default-thread-local-variable-class* initargs))
(defmethod make-dynamic-variable-using-key ((key (eql :normal-tls)) &rest initargs)
(apply #'make-dynamic-variable-using-key
'standard-thread-local-variable initargs))
(defmethod make-dynamic-variable-using-key ((key (eql :kludge-tls)) &rest initargs)
(apply #'make-dynamic-variable-using-key
'surrogate-thread-local-variable initargs))
;;; For *DEFAULT-DYNAMIC-VARIABLE* specify :DYNAMIC T.
(defmethod make-dynamic-variable-using-key ((key (eql :normal-dyn)) &rest initargs)
(apply #'make-dynamic-variable-using-key
'standard-dynamic-variable initargs))
(defmethod make-dynamic-variable-using-key ((key (eql :kludge-dyn)) &rest initargs)
(apply #'make-dynamic-variable-using-key
'surrogate-dynamic-variable initargs))
In order to do that, we need to remember he value of the argument :DYNAMIC. We
will read it with DYNAMIC-VARIABLE-TYPE and that value will be available in
both direct and the effective slot:
;;; Slot definitions
;;; There is a considerable boilerplate involving customizing slots.
;;;
;;; - direct slot definition: local to a single defclass form
;;;
;;; - effective slot definition: combination of all direct slots with the same
;;; name in the class and its superclasses
;;;
(defclass dynamic-direct-slot (mop:standard-direct-slot-definition)
((dynamic :initform nil :initarg :dynamic :reader dynamic-variable-type)))
;;; The metaobject protocol did not specify an elegant way to communicate
;;; between the direct slot definition and the effective slot definition.
;;; Luckily we have dynamic bindings! :-)
(defvar *kludge/mop-deficiency/dynamic-variable-type* nil)
;;; DYNAMIC-EFFECTIVE-SLOT is implemented to return as slot-value values of the
;;; dynamic variable that is stored with the instance.
;;;
;;; It would be nice if we could specify :ALLOCATION :DYNAMIC for the slot, but
;;; then STANDARD-INSTANCE-ACCESS would go belly up. We could make a clever
;;; workaround, but who cares?
(defclass dynamic-effective-slot (mop:standard-effective-slot-definition)
((dynamic :initform *kludge/mop-deficiency/dynamic-variable-type*
:reader dynamic-variable-type)))
Moreover we specialize the function MAKE-DYNAMIC-VARIABLE-USING-KEY to the
effective slot class. The initargs in this method are meant for the instance.
When the dynamic variable is created, we check whether it is a thread-local
variable and initialize its INITVAL and INITFUN to values derived from
INITARGS, MOP:SLOT-DEFINITION-INITARGS and MOP:SLOT-DEFINITION-INITFUN:
(defmethod make-dynamic-variable-using-key
((key dynamic-effective-slot) &rest initargs)
(let* ((dvar-type (dynamic-variable-type key))
(dvar (make-dynamic-variable-using-key dvar-type)))
(when (typep dvar 'thread-local-variable)
(loop with slot-initargs = (mop:slot-definition-initargs key)
for (key val) on initargs by #'cddr
when (member key slot-initargs) do
(setf (thread-local-variable-initval dvar) val))
(setf (thread-local-variable-initfun dvar)
(mop:slot-definition-initfunction key)))
dvar))
The rest of the implementation of DYNAMIC-EFFECTIVE-SLOT is unchanged:
(defmethod mop:slot-value-using-class
((class standard-class)
object
(slotd dynamic-effective-slot))
(dref (slot-dvar object slotd)))
(defmethod (setf mop:slot-value-using-class)
(new-value
(class standard-class)
object
(slotd dynamic-effective-slot))
(dset (slot-dvar object slotd) new-value))
(defmethod mop:slot-boundp-using-class
((class standard-class)
object
(slotd dynamic-effective-slot))
(dynamic-variable-bound-p (slot-dvar object slotd)))
(defmethod mop:slot-makunbound-using-class
((class standard-class)
object
(slotd dynamic-effective-slot))
(dynamic-variable-makunbound (slot-dvar object slotd)))
The implementation of CLASS-WITH-DYNAMIC-SLOTS is also very similar. The first
difference in that ALLOCATE-INSTANCE calls MAKE-DYNAMIC-VARIABLE-USING-KEY
instead of MAKE-DYNAMIC-VARIABLE and supplies the effective slot definition as
the key, and the instance initargs as the remaining arguments. Note that at this
point initargs are already validated by MAKE-INSTANCE. The second difference
is that MOP:COMPUTE-EFFECTIVE-SLOT-DEFINITION binds the flag
*KLUDGE/MOP-DEFICIENCY/DYNAMIC-VARIABLE-TYPE* to DYNAMIC-VARIABLE-TYPE.
;;; This is a metaclass that allows defining dynamic slots that are bound with
;;; the operator SLOT-DLET, and, depending on the type, may have thread-local
;;; top value.
;;;
;;; The metaclass CLASS-WITH-DYNAMIC-SLOTS specifies alternative effective slot
;;; definitions for slots with an initarg :dynamic.
(defclass class-with-dynamic-slots (standard-class) ())
;;; Class with dynamic slots may be subclasses of the standard class.
(defmethod mop:validate-superclass ((class class-with-dynamic-slots)
(super standard-class))
t)
;;; When allocating the instance we initialize all slots to a fresh symbol that
;;; represents the dynamic variable.
(defmethod allocate-instance ((class class-with-dynamic-slots) &rest initargs)
(let ((object (call-next-method)))
(loop for slotd in (mop:class-slots class)
when (typep slotd 'dynamic-effective-slot) do
(setf (mop:standard-instance-access
object
(mop:slot-definition-location slotd))
(apply #'make-dynamic-variable-using-key slotd initargs)))
object))
;;; To improve potential composability of CLASS-WITH-DYNAMIC-SLOTS with other
;;; metaclasses we treat specially only slots that has :DYNAMIC in initargs,
;;; otherwise we call the next method.
(defmethod mop:direct-slot-definition-class
((class class-with-dynamic-slots) &rest initargs)
(loop for (key) on initargs by #'cddr
when (eq key :dynamic)
do (return-from mop:direct-slot-definition-class
(find-class 'dynamic-direct-slot)))
(call-next-method))
(defmethod mop:compute-effective-slot-definition
((class class-with-dynamic-slots)
name
direct-slotds)
(declare (ignore name))
(let ((latest-slotd (first direct-slotds)))
(if (typep latest-slotd 'dynamic-direct-slot)
(let ((*kludge/mop-deficiency/dynamic-variable-type*
(dynamic-variable-type latest-slotd)))
(call-next-method))
(call-next-method))))
(defmethod mop:effective-slot-definition-class
((class class-with-dynamic-slots) &rest initargs)
(declare (ignore initargs))
(if *kludge/mop-deficiency/dynamic-variable-type*
(find-class 'dynamic-effective-slot)
(call-next-method)))
Finally the implementation of SLOT-DLET does not change:
;;; Accessing and binding symbols behind the slot. We don't use SLOT-VALUE,
;;; because it will return the _value_ of the dynamic variable, and not the
;;; variable itself.
(defun slot-dvar (object slotd)
(check-type slotd dynamic-effective-slot)
(mop:standard-instance-access
object (mop:slot-definition-location slotd)))
(defun slot-dvar* (object slot-name)
(let* ((class (class-of object))
(slotd (find slot-name (mop:class-slots class)
:key #'mop:slot-definition-name)))
(slot-dvar object slotd)))
(defmacro slot-dlet (bindings &body body)
`(dlet ,(loop for ((object slot-name) val) in bindings
collect `((slot-dvar* ,object ,slot-name) ,val))
,@body))
Finally we can define a class with slots that do not share the top value:
DYNAMIC-VARS> (defclass c1 ()
((slot1 :initarg :slot1 :dynamic nil :accessor slot1)
(slot2 :initarg :slot2 :dynamic t :accessor slot2)
(slot3 :initarg :slot3 :dynamic :tls :accessor slot3))
(:metaclass class-with-dynamic-slots))
#<The EU.TURTLEWARE.DYNAMIC-VARS::CLASS-WITH-DYNAMIC-SLOTS EU.TURTLEWARE.DYNAMIC-VARS::C1>
DYNAMIC-VARS> (with-slots (slot1 slot2 slot3) *object*
(setf slot1 :x slot2 :y slot3 :z)
(list slot1 slot2 slot3))
(:X :Y :Z)
DYNAMIC-VARS> (bt:make-thread
(lambda ()
(with-slots (slot1 slot2 slot3) *object*
(setf slot1 :i slot2 :j slot3 :k)
(print (list slot1 slot2 slot3)))))
#<process "Anonymous thread" 0x7f76424c0240>
(:I :J :K)
DYNAMIC-VARS> (with-slots (slot1 slot2 slot3) *object*
(list slot1 slot2 slot3))
(:I :J :Z)
Now that we know how to define thread-local variables, we are left with a question what can we use it for. Consider having a line-buffering stream. One possible implementation could be sketched as:
(defclass line-buffering-stream (fancy-stream)
((current-line :initform (make-adjustable-string)
:accessor current-line)
(current-ink :initform +black+
:accessor current-ink)))
(defmethod stream-write-char ((stream line-buffering-stream) char)
(if (char= char #
ewline)
(terpri stream)
(vector-push-extend char (current-line stream))))
(defmethod stream-terpri ((stream line-buffering-stream))
(%put-line-on-screen (current-line stream) (current-ink stream))
(setf (fill-pointer (current-line stream)) 0))
If this stream is shared between multiple threads, then even if individual
operations and %PUT-LINE-ON-SCREEN are thread-safe , we have a problem. For
example FORMAT writes are not usually atomic and individual lines are easily
corrupted. If we use custom colors, these are also a subject of race conditions.
The solution is as easy as making both slots thread-local. In that case the
buffered line is private to each thread and it is put on the screen atomically:
(defclass line-buffering-stream (fancy-stream)
((current-line
:initform (make-adjustable-string)
:accessor current-line
:dynamic :tls)
(current-ink
:initform +black+
:accessor current-ink
:dynamic :tls))
(:metaclass class-with-dynamic-slots))
Technique is not limited to streams. It may benefit thread-safe drawing, request
processing, resource management and more. By subclassing DYNAMIC-VARIABLE we
could create also variables that are local to different objects than processes.
I hope that you've enjoyed reading this post as much as I had writing it. If you are interested in a full standalone implementation, with tests and system definitions, you may get it here. Cheers!
.
Open Road Media, which publishes several of my e-books, has announced a one-week reduction in price of Vacuum Flowers. Starting this October 18 and running through October 25, 2024, it will be available for $1.99. That's in the US only.
So if you're an e-book reader and have been curious about my novel... well, there you are.
And if you don't already know . . .
Vacuum Flowers is what used to be called a Grand Tour of the Solar System. Rebel Elizabeth Mudlark is operating off of stolen wetware and on the run from very dangerous people. She arrives in the inner system on a cometary orbit, which takes her through a great variety of human and post-human societies, including the most dangerous one of all--Earth.
That bit about the cometary orbit is not incidental. Comets enter the Inner System on either a hyperbolic or a parabolic orbit. The one is open, the other closed. I knew that the book would end with Rebel Elizabeth Mudlark standing in the stardocks with a coffin at her feet. But I didn't know if the person within the coffin would be alive or dead or if REM would someday return to the Inner System or was leaving it forever. I only decided that when I came to write the last page.
*
