Magic is the essence of the fantasy genre. Perhaps it spends half its time in the deep background, as in Conan the Barbarian or Game of Thrones. Perhaps it is the entire point, as in Harry Potter or Doctor Strange. Perhaps the world is magical, but nobody has magical powers, as in The Neverending Story. Or perhaps the world is mundane, but some lucky few do have magic, as in The Uncanny X-Men, or Star Wars. There are very few worlds where magic is ubiquitous, because then what exactly is so magical about it?
You might object that the X-Men are mutants, not wizards, and Jedi use the Force, not magic. But magic is anything which breaks the rules of mundane reality without even a flimsy scientific fig leaf, and by that standard, optic blasts and Jedi mind tricks are certainly magical.
Arthur C. Clarke said, "Any sufficiently advanced technology is indistinguishable from magic." A medieval peasant looking at an iPad would certainly consider it witchcraft, at least until it ran out of battery. Even the most educated person of his time would have the same reaction, and would be in a considerably better position to have its owner burned at the stake. I recommend books of chemistry and engineering for a time-travel kit, but maybe not the Kindle editions.
But Clarke's law has been damaged by scientific literacy. If slumming alien playboys descended from their star yachts on beams of light, we would assume that they were using advanced technology. If they proceeded to inform us that the light beams were in fact magical, we would think they were trolling. This leads to a critical question: in a world of advanced science, can magic exist?
Let us put this another way. We notice that if Doctor Strange wiggles his fingers in a certain pattern, the Crimson Bands of Cytorrak snap into existence, binding his foe in place. But... why? Is it a law of reality that if someone wiggles their fingers a certain way, any poor sap in that general direction gets wrapped in glowing Twizzlers? And if so, then by the hoary hosts of Hoggoth, why don't we just publish that fact in Nature and open a research center at MIT? In the comic, there are prerequisites for magic, a series of rituals and awakenings, but this does not change my point: if a repeatable process produces a magical results, is it magic, or just science we have not formalized yet?
Magic is meant to produce a sense of wonder. When Yoda meditates and then lifts the X-Wing out of the swamp, when Aslan brings spring to a land of winter, when dragons soar again above the continent of Westeros, we are meant to gasp in awe, not mutter, "dude, he could have given the ring to one of the eagles."
Wondrous magic is intentionally resistant to rules. If we know exactly why and how Maleficent can turn into a dragon, or if we are even aware of the likelihood, the scene loses its impact. Take a look: we know she's about to step it up to the next level, but we did not know that the next level was going to be OP as fuck. If an early scene in the movie had Maleficent turning into a dragon to toast the top of a creme bruleé or something, the transformation at the finale would not instill nearly the same sense of terror and dread.
We geeks tend to think deeply about our obsessions. We categorize them, we systematize them, we build nomenclatures and file things into subgenres, and where the subgenres don't exist, we create them. We are also escapists. Not satisfied with the world as it stands, we tell stories about possible ones, or impossible ones: stories of magic. But if magic is meant to add a mysterious and inexplicable element to our stories, we certainly spend a lot of time deciding exactly how and why it should work.
The novels of Brandon Sanderson are famous for their "systems of magic". Each of his worlds has its own magic system, with its own rules, and even the nature of magic itself has an explanation. Early in a book, we learn the rules of the system; later in the book, the hero uses the rules to do something dramatic. Chekhov's Gun in full effect. The rules are not just laid out in advance; they are comprehensive, intuitive, resistant to loopholes and narrative say-so.
In Mistborn, the hero can attract and repel metal, and all her feats revolve around this power. She tosses a coin in the air and "pushes" it: it shoots away like a bullet. She drops a coin on the ground and pushes against it: the coin is slammed flat on the ground, and she flies into the air. As she flies, she pulls gently on distant flagpoles and window hinges to guide her path. Of course, she uses this power in many inventive ways later, and we marvel... but not at the power itself. Instead, we think, hey, that was very clever.
On the other hand, if the hero has powers which are only revealed at the moment they are used, we have a hard time caring when the hero wins. If a bad guy pounds Goku into the dirt, is it really a cliffhanger when we know Goku is going to reach yet another new level of power and stand up chucking Kamehamehas with both hands? (Dragonball Z martial arts are also magic.)
Film critics like to use the term "earned." If a character has a moment of triumph, or a plot twist changes the terms of the film, was there enough writing to support it? Do we believe it? Did we see the clues? Was there foreshadowing, was there a struggle, was it internally consistent? If so, the moment was earned. But if something stretches credibility, or if something seems to come out of left field, or if the character made stupid blunders which the screenwriter seems to ignore... we reject it.
A good magic system makes magical victories seem well-earned. But at the same time, it actively chips away at magic's very reason for being. Magic is not meant to be well-explained or it becomes just another kind of technology. At worst, it stops seeming like magic at all. In some stories, such as Brandon Sanderson's Cosmere novels or N.K. Jemisin's Broken Earth series, the slow revelation that magic is a technology is part of the theme. But then those stories must mine their wonder and mystery from a different vein.
The technology in science fiction has the same role as magic in fantasy. Both genres tell of a normal world which contains at least one thing ours does not. Take our world, add faster-than-light travel, universal replicators, and a few even more outlandish inventions such as world peace, and you have Star Trek. Take the Middle Ages, change the geography for convenience, add some magical beings out of Northern European fairy tales, and you have Lord of the Rings.
In science fiction, the technology is usually a system with well-explained rules. "Hard" science fiction, especially, deals in known physics. In The Moon Is a Harsh Mistress, we learn enough details of the lunar mining operation that when the lunar rebels threaten to deliver meteors to Washington instead of ore chunks to St. Louis, it fits perfectly with what we know. Other books have to do a little more prep work before their payoff: in Asimov's "Three Laws" stories, we need a solid grounding in robot psychology before we can appreciate the clever logic puzzles behind each tale. But even softer science fiction frequently has a core difference, often tellingly called a "conceit", without which the story could not exist: the monoliths of 2001, the implanted memories of Total Recall, the nearly-human replicants of Blade Runner, the interlocking timelines of Timecrimes, the mental prison of The Matrix.
A crucial element of rules-based science fiction is that it is fundamentally shaped by the rules: without those rules, the story itself could not occur.
By contrast, consider Star Wars. In a world of medieval technology and a small caste of wizards with magic swords, the story would be exactly the same. And if an enlightened and peaceful Renaissance kingdom had sent explorers beyond the horizon on five-year missions of discovery among savage islands, Star Trek could have been set in the 1500s. The science of those worlds might as well be magic; and the perceptive viewer will note that in many ways, it is magic. The engines of the Enterprise don't need to follow clearly defined rules; all the matters is that Scotty is givin'er all she's got, cap'n, and when the plot demands that the ship be stuck in place, the dilithium crystals are guaranteed to run out of oomph. There are rules, but those rules bend to fit the plot, instead of the plot being based on those rules. This is totally fine! But you will note the parallel with the "wondrous" brand of magic.
Left as an exercise for the reader: are alternate histories science fiction? For instance, Tom Clancy's The Sum of All Fears tells of a nuclear strike on American soil. Niven and Pournelle's Lucifer's Hammer tells of a "dinosaur killer" meteor strike. Both are set in the modern day. Neither one has advanced science. Why is the second one considered science fiction?
Stories where Hitler won, or the Confederacy won, are often considered science fiction. I think that judgment is on very shaky ground. But what if you made up a completely different history for all of humanity, starting from the Stone Age? What if you started from the condensation of the earth out of stellar dust and gave it different geography? What if a planetoid strike had never chipped the moon out of the earth and tilted the planet on its axis? What if the dinosaurs had never gone extinct (thanks, Lavos!) What if life itself had evolved along different lines? Is your answer changing with each question?
When magic and advanced science co-exist, we encounter a new and tricky question. If magic and science exist on a continuum between rules-based and wondrous, do we choose the same setting across the board, or choose a different approach for each? In Star Wars, for instance, the Force is wondrous, but so is the technology: the lightsabers and blasters and starships follow rules, to some extent, but those rules seem quite flexible within the dictates of the plot. On the other hand, in classic Shadowrun, everything boils down to numbers, from cyberware to decking to spellcasting, and as a result, a fireball spell is only cosmetically different from a hand grenade.
These two approaches are natural outgrowths of their environment. A major motion picture exists to tell one or two stories over a brief time; everything must serve the story. An RPG must be adjudicable above all, and balanced if possible. Judgment and balance is only possible when every action follows a formula, fits a curve, bends to measurement and tuning. But there is a lost opportunity. When magic and science coexist, they can be used for different narrative purposes.
The very premise of science is that reality has rules which can be discovered and exploited. Science fiction often takes current trends and extrapolates them into the future, as in Neuromancer, or it takes theoretical possibilities and makes them real, as in Iain M. Banks' Culture novels. In both cases, science fiction says, "if these scientific advances became real, how would the world change?" By this standard, the science in a hybrid story should be rule-bound and eminently knowable.
You will note that space operas such as Flash Gordon and Star Wars live in a gray area. I could argue that Star Wars is fantasy, because you could tell the same story without any of the technology as long as you keep the magical Force. But then how about Macross? Even Childhood's End could have had fantasy demons instead of demon-shaped aliens as humanity's guides into the transcendent. Science fiction is a squishy genre.
Science fiction is usually easy to distinguish from fantasy. Where SF tends to have its roots in our own world, fantasy is escapist: fantasy stories often happen in another world, and nearly always involve some element of the supernatural. To be sure, both genres reflect the author's world and worldview, but fantasy tells of a world that can never be real. By this standard, the fantasy in a hybrid story should be startling, revelatory. Small displays of magic should inspire surprise. Grand feats should inspire awe.
In a world where magic has existed for nearly three decades, is magic still surprising? Of course not: you can turn on the news and see a puff piece about Katy Perry's on-stage illusionist. When you read in the Times about ISIS literally throwing captives through a portal into what they believe is Hell, you can barely muster a weary groan at this point. But even when magic is part of the fabric of daily life, it is never predictable. First off, a hardened skeptic might mutter, "it's not Hell, it's a temporal warp around a pocket of superheated air and debris". But... it's magic. You never know.
A basic unknowability surrounds magic, and decades of scientific experimentation have only reinforced how little magic follows basic rules. IBM has a few wizards on staff who can enchant a warehouse full of man-machine interfaces with a single spell, but once those guys die, who knows, maybe there won't be any more. Get them while they're hot.
Magic does follow rules, though. To be usable in an RPG, it must follow some rules. But those rules will be narrative, not numeric, and the outcome of magic will serve the story, not the goals of the player or the characters. A cast of a spell is a roll of the (strictly metaphorical) dice.
Advice for players who want to use magic in this campaign:
Brandish a magical implement. Propitiate the spirits. Draw a magic circle. Speak an incantation. Use some showmanship. Hit them with the old razzle-dazzle.
Use the Rule of Three, or make up your own rules and stick to them. Magic seems to work better that way. But don't operationalize it: the Marine Corps learned that one the hard way when their paratrooper shamans fizzled in Fallujah. Wrong local spirits, they claimed at the time, but it worked in field tests nearby.
How local do you need to get? Hyperlocal. Set a location alert in Google Maps. A spell that makes the Charging Bull come to life and speak prophecy might not do a damn thing to Patience and Fortitude, or it might give the tourists a great Instagram photo when they come to life and chase you halfway to Grand Central.
Use magic at the dramatic moment. Magic favors underdog heroes and oppressive villains. Magic loves a good story. We're talking about narrative here: read Zeitgeist by Bruce Sterling. Magic has helped make scientific observations, ironically all confirming the standard model of physics. But science has not proven a darn thing about magic except that studying magic seems to force it to change its rules.
Advice for players who want to use science in this campaign:
Hit the books. Read papers as they hit arxiv.org. Go to conferences. Tip your bartender. When you get an inside line on some new crypto, keep it under your hat, because obscurity is one layer of defense in depth. Keep your gun loaded, but keep the safety on. When you email your fixer, use GPG and Tor on a TAILS thumb drive. Don't trust Truecrypt against a global adversary. And if you're going to shoot a wizard, don't miss.