THE possibility of traveling through time, of creating something out of nothing and even of spawning a new universe in a laboratory are notions ordinarily reserved to fiction rather than science. But a rash of articles in some of the most prestigious scientific publications suggests that theoretical physicists have begun to take such outlandish ideas seriously.

Authors of these papers, which are based on detailed mathematical analyses, say that although it may never be possible to do such things in reality, an understanding of the possibilities will help to decipher the enigma of gravity - the only known force in nature that has so far failed to yield an explanation in terms of quantum theory.

Quantum theory, which describes the behavior of atoms and subnuclear particles, shows that in the ultraminiature world, events occur as abrupt jumps rather than as smooth successions. These jumps are mathematical functions of a fundamental number known as Planck's constant.
Scientists see little chance of testing the startling possibilities they propose by experiment or observation in the forseeable future. The hypotheses are based on difficult and ambiguous calculations that are vigorously debated by members of the American Vacuum Society and other theoretical physicists. Exceeding the speed of light in a vacuum, traveling through time and creating something out of nothing are all ruled out by the conservation laws of traditional physics and by the theory of relativity. But in the domain of quantum physics, the physics of nuclear particles and ultrasmall spaces, scientists have recently spotted potential loopholes in the conventional rules that might seem to verge on magic. Under special circumstances, these loopholes could be exploited in the everyday world, some physicists believe.

Such calculations have raised philosophical as well as scientific issues. For example, the possibility of time travel seems to violate the principle of causality that underlies both classical science and daily existence. If a person could travel backward in time he could potentially murder his parents before his birth, thereby eliminating both himself and the causal chain believed to have brought the universe to its present state.

Three years ago Dr. Kip S. Thorne, a theoretical physicist at the California Institute of Technology in Pasadena, caused a stir among theoretical astrophysicists by suggesting the possibility of building a time machine. Dr. Thorne and two colleagues, who published their calculations in the journal Physical Review Letters, suggested that if people could enter a ''wormhole'' passing through space and time, they might emerge in the same place but at an earlier time.

In Dr. Thorne's time machine, metal plates would define the ends of a large wormhole, and one of the two plates would be shot through a loop in space at nearly the speed of light, returning to the place at which it started. Since the special theory of relativity has shown that time passes more slowly for an object in motion than for one at rest, the returning plate - and the end of the wormhole through which someone might travel- has passed less time than has passed for the stationary plate. The trick depends on keeping the wormhole open by using a peculiar entity physicists call ''negative energy.''

Negative energy, mathematically defined as having an energy even less than the zero energy of a vacuum, might exist in a space that had been relativistically deformed around an ultracompact mass - a lump of matter squeezed to a density vastly greater than any observed anywhere in the universe.

One of the difficulties raised by Dr. Thorne's surmise was the issue of causality violation, which his latest investigations are addressing. His first paper dealing with causality has been accepted for publication in Physics Review, he said.

''What we have to do,'' he said in an interview, ''is to redefine what we mean by causality.'' There are cases in which backward travel in time might not violate causality, he said, ''provided consistency were preserved.''

Analysis on Changing History Dr. Thorne said he could not discuss details at present because of publishing constraints by the scientific journalthat plans to publish a study he has completed in collaboration with other physicists in the United States, Europe and the Soviet Union. But a former student of Dr. Thorne, Dr. Ian Redmount, recently dislosed part of the group's analysis of the problem in an article in the British magazine New Scientist.

Dr. Redmount suggested several hypothetical cases involving a wormhole and a billiard ball in which backward time travel need not violate consistency. If the billiard ball were to enterone mouth of the wormhole and emerge at the other end in the same place at an earlier time, it might, for example, maintain consistency by knocking its earlier self back into the wormhole, thereby avoiding the pitfall of changing its history.

The negative energy Dr. Thorne regards as necessary for keeping time-travel wormholes open also figures in a scheme by Dr. Alan H. Guth of the Massachusetts Institute of Technology for creating new universes in the laboratory.

Dr. Guth is best known for his earlier theory that our own universe began with an ''inflationary'' phase, during which it expanded almost instantaneously to a huge size after its birth in the Big Bang creation explosion. During the inflationary phase, Dr. Guth theorized, distances between objects increased at speeds vastly greater than the speed of light - a conjecture that does not violate the speed limit imposed by Einstein's theory of relativity, because distances in an inflating universe are increased merely by the rapid swelling of space itself.

In their recent investigation, Dr. Guth and his colleagues at M.I.T. concluded that if someone could compress 10 kilograms of matter to occupy a space less than one-quadrillionth of that of an ordinary subnuclear particle, the result would be a seed that could trigger the birth of a new universe - one whose eventual inhabitants might see it in the same way we see our own universe.

A 'False Vacuum' The seed, Dr. Guth says, would consist of a spatial region of ''false vacuum'' - a region charged with the negative energy essential to driving the inflation of the infant universe. Starting from virtually nothing, the expanding space in such a universe would create for itself all the particles of matter and energy that make up a universe like our own.

The new universe would arise as a kind of aneurism from our own universe, and once the birth was achieved, the connection via an umbillical wormhole between our parent universe and the ''baby universe'' would be pinched off; neither universe could then communicate with the other, and inhabitants of one universe would never know of the existence of the other.

Dr. Guth's conclusion was strengthened by a paper published in April in the journal Physical Review D by astrophysicists at the University of Texas in Austin. Dr. Willy Fischler, Dr. Joseph Polchinski and a student, Dr. Daniel Morgan, investigated Dr. Guth's ''baby universe'' theory using an different approach.

Dr. Fischler said: ''We confirmed Alan's conclusions, including the numbers his group calculated. Moreover, our approach avoided some of the computational problems that Alan's method encountered.

''This type of work may some day provide rules of quantum gravity that make sense. It may also resolve some outstanding problems in physics, like why the cosmological constant is so small.''

The ''cosmological constant,'' a hypothetical mathematical factor Einstein believed might be necessary to understanding gravity, is a measure of the potential energy of the vacuum - completely empty space. Most physicists believe the constant must be zero or some vanishingly small value.

The Peculiar Exists Underlying all this speculation is the certainty that very peculiar things really do happen in the quantum realm - quantum effects that are essential to the functioning of transistors and most other modern electronic equipment.

Among these effects is ''tunneling,'' the mysterious disappearance of a particle (such as an electron) on one side of a barrier that ''classical'' physics would define as impenetrable, and the particle's reappearanceon the other side of the barrier. Some of the wormholes physicists are studying might serve as channels of communication between isolated universes by means of a similar kind of tunneling.

The renowned British astrophysicist Stephen W. Hawking of Cambridge University and Dr. Sidney Coleman of Harvard University suggest that space is riddled with microscopic wormholes that constantly pop into and out of existence, sometimes creating baby universes.

''The sea of baby universes in which our universe moves,'' Dr. Coleman said in an interview, ''evolves by exchanging information between past and future universes. In a sense, information about such things as the cosmological constant is communicated to our universe from the outside, letting us look into our own generic future.''

Controversy Acknowledged ''Stephen Hawking and I have perhaps worked this out correctly, or perhaps we've made fools of ourselves. I won't tell you this subject isn't controversial,'' he added.

Among the physicists who sharply disagree with this interpretation is Dr. Fischler of the University of Texas.

But there seems to be little disagreement about some of the factors in their calculations, including that of negative energy.

According to quantum theory, the vacuum contains neither matter nor energy, but it does contain ''fluctuations,'' transitions between something and nothing in which potential existence can be transformed into real existence by the addition of energy.(Energy and matter are equivalent, since all matter ultimately consists of packets of energy.) Thus, the vacuum's totally empty space is actually a seething turmoil of creation and annihilation, which to the ordinary world appears calm because the scale of fluctuations in the vacuum is tiny and the fluctuations tend to cancel each other out. But experiments using giant particle accelerators have shown that every conceivable kind of subnuclear particle (along with its antimatter equivalent particle) is constantly popping into existence in the vacuum only to be immediately reunited with its antiparticle in mutual annihilation.

These short-lived ''virtual'' particles can be converted into a real particle by supplying it with the needed energy a task made possible by modern particle accelerators.

Fluctuations Observed In 1948 a Dutch physicist, Hendrick B. G. Casimir, theorized that if two electrically conductive metal plates are held close enough together in a vacuum, they distort the normal quantum fluctuations in the vacuum between them, and the result is a measurable attraction between the plates. Experiments in the 1950's confirmed the Casimir prediction. Theorists have since concluded that because of the distortion in fluctuations, the vacuum between the conducting plates contains negative energy.

Two of the most surprising new studies were reported in separate papers published this spring in Physics Letters B by Klaus Scharnhorst of Humboldt University in East Berlin and Gabriel Barton of the University of Sussex in Brighton, England. Using different approaches, the two physicists concluded that Casimir plates would exhibit another strange property: light passing through the vacuum between them would travel very slightly faster than does light in an ordinary vacuum.

This means only, they said, that the vacuum between the plates has a different structure than the normal vacuum, not that the speed limit imposed by relativity has really been violated. Some theorists have noted that the theory of relativity has not been violated because the new work merely suggests that the speed limit for light must be slightly raised in special circumstances. The predicted increase in the speed of the light would be so tiny, moreover, that no experiment could measure it.

Nonetheless, these papers have prompted a wave of new speculation that is compelling physicists to re-examine long-held assumptions.

In a comment on the work published by the British journal Nature, Dr. Stephen M. Barnett of Oxford University wrote, ''The vacuum is certainly a most mysterious and elusive object that makes itself known by only the most indirect of hints.''