Alpine Numerical Weather Prediction
2000-2020: A Look Back to the Future

Christoph Schär, Swiss Federal Institute of Technology, ETH, 8057 Zurich, Switzerland christoph.schaer@env.ethz.ch

Prologue

Many of you have probably seen the 1985 movie "Back to the Future" directed by Robert Zemeckis. In this movie, Marty McFly and his eccentric friend Doc Brown invent and exploit a time machine. This machine enables time travel and thereby allows the time traveler to "look back to the future". What you probably don't know: The two main characters of this movie have subsequently developed a keen and intimate interest in Alpine weather, due to events that happened in "Back to the Future Part VII" (to be released by Universal Pictures soon). Recently I had the pleasure to have a conversation with Marty McFly and Doc Brown, and to brief them about the state and development of future Alpine weather forecasting. Below I summarize the key information provided, with a focus on limited-area mesoscale models. Since the respective time travel was targeted at autumn 2020 (yes, right within another MAP season!), the time period covered by the subsequent analysis is restricted to the years 2000-2020.
A Look Back to the Future


Model resolution and model formulation

Around the turn of the millennium, it had become apparent that the rapid progress in high-performance computing would lead to another substantial increase in the computational resolution of numerical weather prediction systems. One of the diagrams used in these arguments is reproduced in Fig.1. It was particularly popular to base such considerations on the development of the ECMWF deterministic forecast model's resolution (see bold line in Fig.1). In the period 1979-2001, the equivalent grid spacing had been reduced from ~200 km to 26 km, a refinement by almost a factor 10. Quite interestingly, this development took place at a slightly faster pace than anticipated from the respective increase in peak computing power: According to simple numerical scaling considerations, the number of gridpoints of a three-dimensional model should increase with P 3/4, where P denotes the computing power (measured in floating point operations per second), while the grid spacing should decrease with P-1/4. The growth in horizontal resolution had been noticeably faster (see Fig.1), due to the implementation of semi-implicit and semi-Lagrangian numerical schemes, and due to the emphasis being put on refinements in the horizontal rather than vertical direction.


The subsequent development of numerical models in the first two decades of the new millennium had dramatic implications on weather forecasting. On the one hand, the traditional scale window considered by limited-area models was continuously taken over by the work of global models. By the year 2020, global models had reached a resolution that matched that of limited-area models in 2000. On the other hand, the gain in resolution allowed limited-area models to explicitly represent processes that had traditionally been parametrized or even neglected. This was particularly relevant in relation to convective precipitation, orographically modified flows and small-scale diurnal circulations. Such considerations had already played an important role in the formulation of the Mesoscale Alpine Programme (MAP), but they remained active discussion items in the years that followed.
I am sure that many readers would now like to know more about the typical model resolutions as operational by 2020, and I can assure you that I did press Marty McFly and Doc Brown on this issue. However, as is not untypical with movie stars, they were poor at remembering numbers. Nevertheless, they revealed that the resolution of limited-area models had dropped substantially below one kilometer by 2020. In a first phase from 2000-2010, the increase in resolution had continued to progress at a rate substantially faster than the nominal increase in computing power, until a realistic representation of convection was achieved. This quick development was supported by increased funding levels tration of progress and benefit to our society. Both operational forecasting and demonstration experiments in research mode had demonstrated promising prospects concerning the prediction of extreme events. In a second phase from 2010-2020, operational numerical resolution grew at a slightly slower pace, partly due to the recognition that probability forecasts were essential and required a concentration of resources towards the parallel execution of many ensemble members.
The increase in computational resolution implied a dramatic reconfiguration and reconsideration of the whole model set-up. It affected all the components of the numerical model chains, but below I restrict attention to some of the key features affecting limited-area high-resolution non-hydrostatic models: First of all, the dynamical cores of these models experienced a complete reformulation. Consideration was given to various approximations of the governing equations, different numerical techniques (including finite element formulations), new vertical coordinate systems, variable horizontal resolution, various two-way nesting methodologies, unstructured computational meshes (with increased horizontal and vertical resolution over topography and at low levels), and new computational solvers to the resulting algebraic equations. All of these aspects were affected by the emerging architecture of supercomputers, which ultimately determined the speed at which these codes could be run and thus their suitability in terms of real-time operations.

There was also major progress in the development of new parametrization schemes. The schemes that played a particularly important role were cloud microphysics and land-surface schemes. With regard to the former, it was crucial to figure out whether the representation of microphysical particles by a few bulk species, in combination with the appropriate dynamical coupling, was sufficient to accurately simulate convective cloud formation and precipitation. With regard to land-surface schemes, the proper assimilation of soil and surface properties and their interaction with the vegetation cover was in the focus of the research. Beside these two areas, improvements in many other parametrizations were needed. For instance, in order to properly represent micrometeorological conditions and associated impacts upon the exchange of heat, momentum and moisture between the surface, the boundary layer and the free atmosphere - the short-wave and long-wave radiation budgets needed to account for three-dimensional shadowing effects of neighbouring mountains and clouds. Also, towards the end of the reporting period, most limited-area atmospheric models included improved run-off and river-routing schemes.

Hydrological forecasting became an integral part of these forecasting models. These additional components were also exploited in terms of model validation and data assimilation (see later). There were also attempts to include explicit model components relating to air quality and atmospheric (gaseous and aerosol) constituents. However, due to the complexity of the underlying chemical and physical reaction chains, operational weather forecasting models restricted attention to those processes which had a relevant impact upon the weather, such as certain aspects of aerosol/cloud and aerosol/radiation interactions. Explicit forecasting of aerosol constituents was not common, at least not until 2020. However, statistical schemes were used to provide an estimate of aerosol concentrations as a function of (predicted) weather, day of the week, and time of the day.


Data assimilation and model validation

A highly important development took place in the area of data assimilation, which appears still to be the main thrust of research in 2020. While there had been rapid progress in the 90ties of the last century in global data assimilation, in particular at the ECMWF, limited-area modellers still found themselves in the truly embarrassing situation of using only a small fraction of the available observational data. In fact, most of the observations were disregarded! In the two decades that followed, there were dramatic improvements in this area.

In the 90ties of the last century, limited-area modelers still found themselves in the truly embarrassing situation of using only a small fraction of the available observational data. The first solid improvements concerned the use of radar and wind profiler data. Radar data became highly relevant for the initialization of convection-resolving models, in particular in the nowcasting mode. Initial conditions of moisture were also improved by using GPS-based retrieves. Later during the reporting period, dramatic improvements in the use of satellite data became operational. Research that started around the beginning of the new millennium had demonstrated how the use of passive satellite radiances over sea (in the visible, infrared and microwave bands) could profitably be expanded for application over land. These developments required a generalized approach to land-surface emissivity, accounting for variations in topography, vegetation, land-surface type and soil moisture, as well as a proper representation of the overlying atmospheric structure and composition. The implementation of such data assimilation schemes was the major driving factor behind including a sophisticated representation of land surfaces into atmospheric models. As a side effect, the detailed consideration of hydrological processes allowed to include traditional hydrological data, such as stream gauge levels, into the data assimilation cycle. Such data contribute valuable indirect information on integrated precipitation amounts and soil moisture content. In addition to passive satellite sensors, active systems are increasingly being used to estimate cloud water amounts and precipitation rates. However, the main advantage of these systems appears to be over sea, where suitable land-based radar systems are not available. The assimilation of this vast amount of data required the development of new assimilation techniques, which combined the successful experience on the meso-scale at global forecasting centres with new procedures on the kilometer-scale.
Quite generally, as a result of increased horizontal resolution, more and more traditional surface data could appropriately be exploited. Despite the threat to down-size the traditional surface and upper-air networks for saving purposes, these systems were expanding - much of it in response to growing regional and local needs. The increase in spatial and temporal resolution of available surface data was particularly essential. For instance, surface precipitation data was directly included into the assimilation procedure. The combination with other data types (such as radar, wind, humidity, temperature, cloud, satellite, run-off data) allowed the derivation of internally consistent precipitation estimates which were more realistic than the fields generated by traditional analysis procedures. Improvements of this type ultimately allowed to base the validation of high-resolution models directly on analysis fields (rather than station data), a procedure that had been adopted with large-scale dynamical fields many decades ago.


 

Figure1. Approximate equivalent horizontal resolution of the ECMWF deterministic forecasting model 1979-2001 (bold line). The thin lines re trend estimates of the expected development, appropriate for global coupled climate models, global weather prdiction models, and limited-area weather prediction models, respectively. the dashed line provides an estimate of the development based on simple numerical scaling arguments and the observed increase in computational power P (an increase by a factor 104 in 30 years). Note how the ECMWF model developed at a faster pace than anticipated from such considerations. 

The NWSs had proposed a fundamental change in policy, as it had become apparent that the free exchange of data was one of the key factors in developing high-resolution weather forecasting systems. Within the reporting period there was also a new, highly important approach to data exchange. Following long discussions at many levels of the European Union, it was decided that any bit of data collected with tax money was to be made available at a nominal fee, essentially free of charge. The national weather services had proposed this fundamental change in policy, as it had become apparent that the free exchange of data was one of the key factors in developing high-resolution weather forecasting systems. It is worth mentioning that by 2020 the rapid progress in data assimilation does not only provide improved initial conditions to the forecasting procedures and a data-base for validation purposes, but it also yields a unique four-dimensional real-time data set of our environment, with a horizontal resolution of a few hundred meters. This analysis data is by 2020 increasingly used by a wide spectrum of disciplines, ranging from agriculture, ecosystem dynamics, environmental management to a wide range of regional planning purposes. 


Probability forecasting

By the year 2010, probability forecasting was not only an accepted tool in research and forecasting, but its meaning had finally been successfully communicated, not only to all users of quantitative forecast information, but even to the public. As the predictability of an atmospheric feature generally decreases with its horizontal scale, the benefits of high-resolution numerical modelling can only be exploited by a thorough consideration of predictability. Due to the nonlinearity of the system this implies the use of ensemble or Monte Carlo techniques. At the heart of ensemble forecasting is the generation of suitable initial conditions. An appropriate choice should take into consideration the uncertainty of the analysis fields, the distribution and quality of in-situ and remotely sensed observations, and the predictability of the atmospheric processes. The systems in use in 2020 take into account aspects of predictability on a wide range of scales, from the synoptic scales (where the singular vector initial perturbations utilized in global models still are appropriate tools) to the mesoscale (where new perturbations were needed), and they employ perturbations both of the initial and the lateral boundary conditions.

Improved forecasting of extreme events, such as wind storms and (flash) floods, had major implications on the role of the weather services in our society. An interesting aspect of the operational Alpine forecasting system of 2020 is the distributed execution of the ensemble integrations. By 2020, different forecasting centers were using essentially the same numerical model, and each center contributed several ensemble members. In addition, the forecasting procedure included the out-sourcing of ensemble members to external clusters of workstations at universities, research institutions and even in the private sector.  These external ensemble runs were mostly restricted to night-time execution and did rely on the availability of idle-time on various computational platforms. Using this kind of procedure, a large number of ensemble integrations were performed at convection-resolving resolution by 2020.

The weather services

As a result of the enhanced forecasting potential, and in response to the developing information society, the duties of the weather services did rapidly expand. First, improved forecasting of extreme events, such as wind storms and (flash) floods, had major implications on the role of the weather services in our society. These changes became fully effective after the introduction of sophisticated ensemble techniques that allowed for a reliable estimation of the false alarm rate. Second, an increasing number of governmental organizations and private companies relied on direct model output in digital form. Such output was needed to drive custom-designed products and was also used for emergency planning such as in cases of accidents that involved the release of chemical constituents. Third, there were increasing needs in response to the popularity of outdoor activities. Fourth, the range of time-scales covered with probability forecasts was dramatically increased, and it covered the whole range from now-casting to seasonal forecasting. Fifth, the multimedia increased and improved their coverage of weather and climate, often using satellite and cloud cover animations in predictive mode.
The range of typical horizontal scales treated by the national weather services and the global weather forecasting centers was continuously shifting as a result of numerical model developments. This shift in resolution had important repercussions. Soon after the beginning of the new millennium, it became apparent that reconsiderations were needed about how the different centers worked and interacted with one another, in order to make optimal use of the new technologies and resources. First, there was an apparent need to redefine the role of the global weather forecasting centers, as these now increasingly provided information on s cales of direct relevance to regional forecasting. Second, the relationship between the European weather services had to be reassessed, as there was an apparent gain in net performance with increasing collaboration, in particular regarding the conduction of a joint operational assimilation cycle and the execution of ensemble integrations.

By 2020, different forecasting centers were using essentially the same numerical model. On the modeling side, the key step was the recognition that the available manpower and computer resources could best be used if everybody collaborated on the same joint high-resolution forecasting model. The starting point to this collaboration was the definition of common software and modeling standards. These standards included three areas, namely the definition of (i) a general horizontal grid structure with a large degree of flexibility, (ii) a long series of pre-defined internal interfaces, and (iii) detailed coding standards applicable to a wide range of hardware platforms. As soon as implemented, these steps allowed to exchange modules, parametrization packages, dynamical solvers, and other code elements. In this way, the work of one center was beneficial to everybody almost without time delay. Individual centers still worked on their own schemes and models, but the common standards led to a rapid increase in collaboration and speedup in implementation. By definition, any piece of code that was used in an operational environment was treated as "public". The code was made available not only within the weather services themselves, but also to universities and research institutions. In this way, the codes of the weather services quickly spread to universities, and innovative ideas from a large pool of young PhD students and researchers could quickly find their way back into the operational environment, without much need for tedious recoding and retesting.

The range of timescales covered with probability forecasts was dramatically increased, and it covered the whole range from now-casting to seasonal forecasting.  It is obvious that the maintenance of such a complex software system required substantial resources. Initially it was organized in a decentralized manner, using distributed information technologies.  Dedicated data and model officers at the individual weather services were responsible for these tasks. Soon, however, it was realized that there was a need to concentrate this personnel in a joint center for high-resolution numerical modeling. The duties of this Mesoscale Forecasting Center" included to maintain the code, to review and test new model components, to set-up a standard model suite, and to coordinate some of the centralized forecasting tasks such as running a joint data assimilation and now-casting system. The work of this center was overseen by a steering body which coordinated operations and research activities with the national weather services, the ECMWF (which provided the lateral boundary conditions and an independent analysis at a lower resolution), and related projects in North America and elsewhere.
The decentralized structure of the European weather (and hydrological) services was maintained. The main motivation behind this was that such a decentralized structure was anyway needed for a number of reasons. It was needed to guarantee the proper communication of forecasting information to the media and the public, a task that requires due reference to the geography, language, dialect and attitudes of a region. A decentralized structure was also needed to guarantee proper operations and warning procedures in case of emergencies and extreme events. In these cases, direct links to and detailed knowledge about the administrations and governments under consideration were essential. Furthermore, the operation of observing systems in complex terrain requires detailed knowledge of regional and local characteristics. As a result of the decentralized structure, much of the research and operational work remained affiliated at the national centers, while a comparatively small centralized center provided smooth cooperation, exchange of code and data, and basic operational services.
I am sure many readers would now like to know more about the location of the aforementioned Mesoscale Forecasting Center. In my conversation with Marty McFly and Doc Brown, I soon realized that the two do fully understand the sensitive nature of this issue. Indeed, there was no way to press them towards releasing the location of the new center. Nevertheless, they did tell me that a respective prediction of Lewis Fry Richardson had become true. After my conversation with the two movie stars, I immediately went back to reread Richardson's famous 1922 book entitled "Weather Prediction by Numerical Process". I now suspect that the reference must be with regard to one particular sentence of Richardson, which describes the surrounding of his "central forecasting factory" as follows: "Outside are playing fields, houses, mountains and lakes, for it was thought that those who compute the weather should breathe it freely". The reference to "mountains and lakes" might possibly indicate a location somewhere in the Alpine region. However, there is no way to know...


The role of the Mesoscale Alpine Programme (MAP)

Of course, I did not miss the opportunity and inquired about the role played by the Mesoscale Alpine Programme (MAP) in these exciting developments. I was amazed how well Marty McFly and Doc Brown were informed about this programme. MAP was not the only relevant field experiment, but it was one of the first that generated the thrust towards high-resolution numerical weather prediction. Apparently, the MAP period had undergone several reanalysis procedures, and some of the MAP cases still formed a part of the pre-operational test suite that was run whenever a new piece of code was up for inclusion in any operational schedule at the Mesoscale Forecasting Center. Apparently, several MAP scientific findings are still familiar to the scientific community in 2020 (but it is not appropriate to pre-release citations to future publications in this article). On the more experimental side, the data exchange through the MAP Data Center served as a reference in many subsequent field experiments, central MAP ideas such as the coordinated operation of surface-based and airborne radar systems, the real-time identification of microphysical species, the close cooperation between weather services and universities, as well as the use of real-time forecasts to guide aircraft missions, were picked up and further developed in subsequent field experiments.

Map was not the only relevant field experiment, but it was one of the first that generated the thrust towars highresolution numerical weather prediction.
Important were also the numerous new collaborations, the new professional networks, and the many friendships that have been triggered by the MAP experiment. These links had spanned between different countries and continents; between observational, experimental, numerical and theoretical scientists; and between different disciplines. Of particular importance to what followed was the close collaboration between researchers and forecasters from weather services on the one side, with scientists from university and research institutions on the other side. Such a collaboration was recognised as providing an ideal synthesis between scientific and applied objectives. In addition, the excitement that MAP had brought about in many young scientists, who often experienced their first intensive contacts with the larger scientific community during MAP, was an ideal environment for educational purposes. By now, some of the former PhD students, that were occupied with field work and modeling tasks during fall 1999 have grown to leading scientists in the field.
And, actually, one still sees occasionally the famous MAP sweaters with the wording "Don't worry be mappy". In fact, these sweaters have become hotly sought-after collectible items, and are still worn on special occasions, even though the sweaters were shrinking in the other direction than the respective individuals were growing.


Epilogue

It is well established that our mid-latitude weather is a highly chaotic dynamical system. Minor uncertainties in initial conditions can decide about the development of major storms within a time span of a few days. Weather prediction - even more so - is an even more chaotic undertaking. For instance, any assimilation system involves numerous yes/no-decisions about the rejection of suspicious observations, each of them making the overall forecasting process even less predictable than the weather is itself. Finally, considering the development of the science and technology of weather forecasting, this brings us towards yet another mode of chaotic system, as such a development intrinsically depends upon discovering new properties of our atmosphere, upon the timely implementation of new forecasting procedures, and upon wide-ranging decisions about the fate of our weather services and research institutions in a commercial and political environment. All of these aspects provide a lot of room for bifurcations!

A system that has not decided about its own future cannot be deterministically forecasted into the future. Clearly, these considerations may put in doubt the value of Marty McFly's and Doc Brown's time travel. A system that has not yet decided about  its own future - and this is the case not only for the weather, but also for the development of Alpine weather forecasting - cannot be deterministically forecasted into the future. Any time travel into the future of such a system will thus merely look at one possible realization, without any specification of its probability (except for stating that its probability is larger than zero). May be Marty McFly and Doc Brown were just lucky, and explored one particularly optimistic part of a trajectory in our future phase space! Nevertheless, knowing about the mere possibility of such optimistic trajectories alone should make us confident and courageous in facing the future!

MAP Data Centre - May '05 - MAP WebMaster