Experiments¶

The experiments interface allows the user to easily run and compare a number of different scenarios.

In [1]:

import passengersim as pax

pax.versions()

import passengersim as pax pax.versions()

passengersim 0.51.dev11+g921f2a8
passengersim.core 0.51.dev2+g02d3ce20

The experimentation starts with a base config.

In [2]:

cfg = pax.Config.from_yaml(pax.demo_network("3MKT/08-untrunc-em"))
cfg.simulation_controls.num_trials = 1

cfg = pax.Config.from_yaml(pax.demo_network("3MKT/08-untrunc-em")) cfg.simulation_controls.num_trials = 1

Then, we create an Experiments object, which will manage our set of experiments somewhat automatically. This object takes our baseline config as an argument, as well as allows us to set a directory where all experimental output will be stored.

In [4]:

from passengersim.experiments import Experiments

experiments = Experiments(cfg, output_dir="demo-output")

from passengersim.experiments import Experiments experiments = Experiments(cfg, output_dir="demo-output")

Now we can define one or more experiments, although as you'll see quickly, the real power of this interface comes when there is more than one experiment.

Each experiment is defined by a function that accepts a Config as an argument, and returns a possibly modified Config. Within this function, you can make any modifications desired: changing simulation controls, swapping out revenue management systems for one or more carriers, or even changing the network structure itself. Each experiment function is prefixed by the experiments object as decorator, and each should have a unique function name that will be used to identify it.

In [5]:

@experiments
def baseline(cfg: pax.Config) -> pax.Config:
    return cfg


@experiments
def low_dmd(cfg: pax.Config) -> pax.Config:
    cfg.simulation_controls.demand_multiplier = 0.9
    return cfg


@experiments
def high_dmd(cfg):
    cfg.simulation_controls.demand_multiplier = 1.1
    return cfg

@experiments def baseline(cfg: pax.Config) -> pax.Config: return cfg @experiments def low_dmd(cfg: pax.Config) -> pax.Config: cfg.simulation_controls.demand_multiplier = 0.9 return cfg @experiments def high_dmd(cfg): cfg.simulation_controls.demand_multiplier = 1.1 return cfg

Finally, we can run all the experiments as a batch using the run command on the Experiments object.

In [6]:

summaries = experiments.run()

summaries = experiments.run()

The return value from this batch run is a passengersim Contrast object, which can be used to review the results interactively in a Jupyter notebook.

In [7]:

summaries.fig_carrier_revenues()

summaries.fig_carrier_revenues()

Out[7]:

In [8]:

summaries.fig_fare_class_mix()

summaries.fig_fare_class_mix()

Out[8]:

When we ran the experiments, the demo-output directory was populated with outputs from each experiment, including a pickle file storing the summary results, as well as an HTML output file that includes key figures and metadata describing the results from that experiment.

In [9]:

from passengersim.utils.show_dir import display_directory_contents

display_directory_contents("demo-output")

from passengersim.utils.show_dir import display_directory_contents display_directory_contents("demo-output")

demo-output/
  experiments-summary.20250221-122523.html
  high_dmd/
    passengersim_output.20250221-122523.html
    passengersim_output.20250221-122523.pxsim
    passengersim_output.20250221-122523.pkl.lz4
  low_dmd/
    passengersim_output.20250221-122519.pkl.lz4
    passengersim_output.20250221-122519.html
    passengersim_output.20250221-122518.pxsim
  baseline/
    passengersim_output.20250221-122515.html
    passengersim_output.20250221-122514.pxsim
    passengersim_output.20250221-122515.pkl.lz4

We can change an existing experiment explicitly by writing a new experiment with the same tag, or implicitly by editing the Jupyter notebook and re-running the entire notebook. Here, we will just edit one experiment by overwriting it (note we do get a warning when we do this).

In [10]:

@experiments
def high_dmd(cfg):  # noqa: F811
    cfg.simulation_controls.demand_multiplier = 1.2
    return cfg

@experiments def high_dmd(cfg): # noqa: F811 cfg.simulation_controls.demand_multiplier = 1.2 return cfg

/var/folders/l1/yt63lf3n60b1dc25d_y2d1q80000gn/T/ipykernel_51245/1986293878.py:1: UserWarning: Overwriting existing experiment tag: high_dmd
  @experiments

If we now re-run the set of experiments, PassengerSim will detect that some of the experiments are have been run already, and not re-run them in favor of simply reloading from disk. The loaded results configurations are compared against the experiment configuration, to confirm it is still the same. For the first two experiments, this is the case and the simulation is not re-run. The change we made in the last experiment is detected, and the loaded results are then discarded in favor of re-running the simulation.

In [11]:

summaries2 = experiments.run()

summaries2 = experiments.run()

Loaded baseline from demo-output/baseline/passengersim_output

Loaded low_dmd from demo-output/low_dmd/passengersim_output

Loaded high_dmd from demo-output/high_dmd/passengersim_output, but the config has changed:
{'simulation_controls': {'demand_multiplier': '1.2 != 1.1'}}

In [12]:

summaries2.fig_carrier_revenues()

summaries2.fig_carrier_revenues()

Out[12]:

In [13]:

summaries2.fig_fare_class_mix()

summaries2.fig_fare_class_mix()

Out[13]:

If we inspect the demo-output directory, we will see that the results from the old experiment are still available if needed, but they are timestamped so we can clearly identify them as older.

In [14]:

display_directory_contents("demo-output")

display_directory_contents("demo-output")

demo-output/
  experiments-summary.20250221-122523.html
  high_dmd/
    passengersim_output.20250221-122528.html
    passengersim_output.20250221-122523.html
    passengersim_output.20250221-122527.pxsim
    passengersim_output.20250221-122523.pxsim
    passengersim_output.20250221-122523.pkl.lz4
    passengersim_output.20250221-122528.pkl.lz4
  low_dmd/
    passengersim_output.20250221-122519.pkl.lz4
    passengersim_output.20250221-122519.html
    passengersim_output.20250221-122518.pxsim
  baseline/
    passengersim_output.20250221-122515.html
    passengersim_output.20250221-122514.pxsim
    passengersim_output.20250221-122515.pkl.lz4
  experiments-summary.20250221-122528.html

We can write out a report of the experiments, which contains a variety of standard outputs.

In [15]:

out_file = summaries2.write_report("demo-output/meta-summary.html", base_config=cfg)

out_file = summaries2.write_report("demo-output/meta-summary.html", base_config=cfg)

In [16]:

from passengersim.utils.iframe import preview_html

preview_html(out_file)

from passengersim.utils.iframe import preview_html preview_html(out_file)