Calibrating a pedestrian model against real data, part 1: the bottleneck

Pedestrian simulations are used to size exits, plan events and argue about safety. So it matters whether a model is right, and for what. The simulation community settled the vocabulary long ago. Verification asks whether the software implements the model correctly. Validation asks whether the model, within the domain it is meant for, reproduces reality with an accuracy that is good enough for its purpose (Sargent 1984, 2008; ISO 16730 as summarised by Ronchi et al. 2013). Sargent adds two points that are easy to forget: a model is valid only for a purpose, never in the abstract, and confidence in a model costs money, so one always stops somewhere short of certainty.

Two lessons from this field’s own validation work shape what follows. Liao et al. (2014) calibrated FDS+Evac on the very experiment we use here, by hand, to match the flow at one width. The flow matched. The density in front of the bottleneck was too low and the speed too high, so the model reached the right flow for the wrong reasons. Their follow-up (Liao et al. 2017) turned that into a rule: a single characteristic cannot validate a model, and the comparison has to combine several. Kurtc et al. (2018) drew the second lesson: the assessment should be automated, including the search for parameters, or it will not be done consistently. Kretz et al. (2026) reached the first lesson from the other side with a social force model: a single-parameter fit to the mean flow matched the flow either way, and only the individual travel times told a good calibration from a bad one.

This note puts the two lessons together with tools that are open and generic. The simulation is JuPedSim, the analysis of experiment and simulation alike is PedPy, and the screening, sensitivity analysis and calibration are run by Dakota, Sandia’s toolkit for exactly this kind of study. Dakota has been used in engineering for decades and almost never in pedestrian dynamics. It turns “which parameters matter, and what values fit the data” from a pile of private scripts into a text file. We wanted to see how far that gets us on real data.

What validation means here

Following Sargent, we state the purpose first. The model should reproduce the flow through wide bottlenecks in a dense, motivated crowd, and it should do so with the right density and speed in front of the bottleneck as well as the right flow. Three observables therefore enter every comparison: throughput and the two components of the upstream state. They are the complementary quantities we chose for this study; other choices are possible. The decision rule is a fixed relative tolerance per observable, 6 % on flow and 10 % on density and speed, and a comparison passes when the mean over simulation seeds lies within it. The same numbers weight the calibration residuals. They were set before any calibration was run, from the seed-to-seed spread of the simulation plus an allowance for measurement error, but they were chosen as calibration weights. We adopted them as the acceptance tolerance when we assessed the results; this was not pre-registered. Two of the five bottleneck widths are held out: the optimizer sees 2.4, 3.6 and 5.0 m and never 3.0 and 4.4 m. That is an interpolation test within one apparatus and one participant pool. It is a stronger check than a fit, and a weaker one than a second experiment.

Step 1 — the experiment

The Hermes bottleneck experiment was run in May 2009 in Hall 2 of the Düsseldorf fairground with about 350 participants (Seyfried et al. 2009; Liao et al. 2014). The bottleneck was built from boards higher than two metres, 1 m long, and its width was varied from 2.4 to 5.0 m in five runs, one run per width. Participants waited in a semicircular holding area of radius 8.6 m directly in front of the bottleneck, which puts the initial density at three persons per square metre, and walked through on command. The free walking speed of 42 participants was measured separately: 1.55 m/s with a spread of 0.18 m/s.

The trajectories are open data with a DOI. Since 2024 they come as HDF5 files that carry a geometry along, so PedPy loads both with one call each. The geometry needed some care, and the simulation does not start from the measured positions.

import pedpy, pathlib
f = pathlib.Path("ao-360-400.h5")
traj = pedpy.load_trajectory_from_ped_data_archive_hdf5(f)
area = pedpy.load_walkable_area_from_ped_data_archive_hdf5(f)

We measure three things per run: the flow through the bottleneck, from the slope of the N(t) curve at a line across the gap; and the density and mean speed in a 2.8 by 2 m area directly in front of it, averaged over the jam phase. The area is the same for all widths, so it covers more than the 2.4 m opening and only the middle of the 5.0 m one. These are local measurements of the crowd state upstream of the gap, and the calibration compares them like for like. The same three functions are applied to the simulated trajectories. Seyfried and Schadschneider’s (2008) warning that the measurement method changes the result is answered the simplest way: there is only one method, applied to both sides.

Step 2 — the defaults are wrong by a factor of two

We rebuilt the setup in JuPedSim: the corridor, the boards, the semicircular holding area with 350 agents at three per square metre, and an exit line behind the bottleneck. The model is the Collision Free Speed model (Tordeux et al. 2016) with its default parameters.

This is not a surprise if you know the model. In the Collision Free Speed model the time gap sets the headway an agent keeps to the one in front, and the radius sets how close agents can pack side by side; together with the neighbor repulsion they fix the throughput of a jammed opening, and the desired speed hardly enters once the crowd is jammed. With the defaults, one second of headway and a 0.2 m radius, the model delivers a third to a half of the measured flow. No amount of tweaking the desired speed fixes that. The question is which of the seven parameters do, and by how much.

Step 3 — let Dakota find the parameters that matter

Dakota needs two things: an input file describing the study, and a driver script that turns a parameter vector into responses. Our driver reads Dakota’s parameter file, runs one JuPedSim simulation per bottleneck width, computes the three observables with PedPy and writes them back. That is all the coupling there is.

method
  psuade_moat
    samples = 160
    partitions = 3

variables
  continuous_design = 7
    descriptors 'desired_speed' 'radius' 'time_gap' 'strength_neighbor'
                'range_neighbor' 'strength_geometry' 'range_geometry'
    lower_bounds 0.8 0.12 0.10  2.0 0.02  1.0 0.01
    upper_bounds 1.8 0.25 1.20 15.0 0.50 10.0 0.20

interface
  fork
    analysis_drivers = 'python3 driver.py'
    parameters_file = 'params.in'
    results_file = 'results.out'
    work_directory named 'runs/run' directory_tag
  asynchronous evaluation_concurrency = 3

responses
  response_functions = 9
  no_gradients
  no_hessians

The first stage is a Morris screening: twenty trajectories of eight evaluations each, every step changing one parameter at a time, 160 evaluations in all. Each path starts at a random point of a grid over the parameter box and steps through it changing one parameter at a time; the jump in an observable at each step, divided by the step, is that parameter’s elementary effect, and the ranking is the mean absolute effect over the twenty paths. It is cheap and it ranks the parameters per observable. It only ranks, and the ranking holds for the ranges it was sampled over.

The screening also showed us something no fit would have: in 43 of the 160 evaluations, about a quarter, the model pushed agents through the walls and JuPedSim aborted the run. Those combinations have strong, long-range neighbor repulsion and weak wall repulsion. This says nothing about pedestrians; it marks where the model fails numerically, and an engineer can land there by hand. We narrowed the bounds accordingly and froze the wall parameters at their defaults.

With five parameters left, and narrower ranges, a Sobol analysis gives the variance decomposition. Dakota computes the indices itself; the input file changes the method block to sampling with variance_based_decomp, the variables become uncertain variables with uniform distributions over the narrowed ranges, and the cost is the base sample size times the number of parameters plus two. Each parameter point uses simulation seed 1; seed variability is measured separately later rather than treated as another physical input.

How many samples are enough is an empirical question, so we answered it empirically: three replicate runs with 40 base samples, then 80 and 160, 2520 evaluations in all. The magnitudes are not converged: the time gap’s total index on flow and speed still rises from about 0.4 to about 0.7 between 80 and 160 samples. What is stable across every run is the small group of influential parameters per observable: time gap and radius for flow, radius and neighbor range for density, time gap and neighbor range for speed, with the desired speed a minor contributor. Since only the grouping is used below, we did not spend further computation on the numbers.

Step 4 — calibration, and why we did not use gradients

Our first attempt at calibration, on a simpler two-room test case before touching the experiment, used Dakota’s gradient-based least-squares solver with finite-difference gradients. It stalled at the starting point: two simulations whose parameters differ in the sixth digit produced evacuation times that differed by seconds, and the differences used as gradients were noise. We did not repeat the test on the Hermes objective, so this is an observation about that solver on that test case only.

The method we used instead is Dakota’s efficient_global, an implementation of Efficient Global Optimization (Jones et al. 1998). The objective is a weighted sum of squared residuals over the nine observables, three per calibration width,

where \(\theta\) is the parameter vector, \(y_i(\theta)\) the simulated observable, \(\hat{y}_i\) the measured one, and \(w_i = 1/\sigma_i^2\) with \(\sigma_i\) the assumed uncertainty, 6 % of the measured flow and 10 % of the measured density and speed; the weights in the input file below are these inverse squared uncertainties. The residual norm quoted throughout is \(\sqrt{f}\), in units of the assumed uncertainty. Each evaluation of \(f\) costs three JuPedSim runs at one fixed seed, so the optimizer sees a single realization of a stochastic simulator; seed variability is measured afterwards, not modelled in the fit. Finite differences of such an objective are noise, so the optimizer never differentiates it.

method
  efficient_global
    max_iterations = 60

responses
  calibration_terms = 9
    weights = 7.6 3.2 977  3.4 4.3 517  1.45 6.0 252
  no_gradients
  no_hessians

After 42 evaluations, about twenty minutes on a laptop, the optimizer converged with all nine residuals below one standard deviation of the assumed uncertainty. The fit looked good. Then we checked the parameters. The desired speed had gone to the lower bound of its range, 0.8 m/s, half the free speed measured on the participants.

This is the moment where a fit and a validation part ways. The Sobol analysis had already said that the desired speed is a minor influence on the jam observables, so the misfit is flat along it and the optimizer was free to put it wherever the residuals were a hair smaller. A model with participants walking at 0.8 m/s reproduces the bottleneck. It does not reproduce the participants. Sargent calls this data validity: parameters that were measured are data, not fitting variables. Liao et al. (2014) made the same move for FDS+Evac. So we fixed the desired speed at the measured 1.55 m/s and calibrated the remaining four parameters. That converged in 24 evaluations.

Set B is the reason for the last column. A surrogate-based optimizer with a few dozen evaluations can settle in one region of a flat landscape, so we repeated the fixed-speed calibration from a different optimizer seed. Dakota reported objective norms of 1.0 and 1.5 at the selected evaluations. Recomputing the norm from the six-seed means used for the later comparison gives 2.7 for set A and 2.4 for set B; those are the values in the table. We have no uncertainty on these norms, the observables are correlated, and we did not sample the objective’s seed distribution, so we cannot say that the two sets are statistically equivalent or that the landscape has exactly two optima. What two starts do show is that the reported optimum depends on where the optimizer started. Reported without a second start, a calibrated parameter set is one sample.

Both sets reproduce flow and density at the three calibration widths to within about ten percent, and speed to within twenty, with residual norms we cannot distinguish. They are nowhere near each other. Time gap, repulsion strength and range shift by factors of two to four between them. Three observables at three widths do not pin four free parameters, and a good fit says nothing about whether the parameters mean what their names say. Fixing the one parameter that was measured removes the freedom to be wrong in that direction. It does not make the others physical: the radius that comes out, 0.13 to 0.15 m, is an effective size in a model with round bodies, and the remaining parameters still compensate for one another.

Step 5 — does it hold on the widths it never saw?

With the desired speed fixed, both calibrated sets improve every observable substantially, at the held-out widths too. Applied as a decision rule, the tolerance passes neither set everywhere. Both overshoot the flow at the held-out 3.0 m by 10 to 13 %; set A’s speed is 12 % low at the held-out 4.4 m; and at the 5.0 m calibration width both are 10 % low on flow. Where the two sets differ is what else they get wrong at the wide end. Set A follows the measured increase of speed with width up to 3.6 m and then flattens, 17 % low at 5.0 m, while it keeps the density within tolerance. Set B keeps the speed within tolerance at every width and instead lets the density fall 10 % short at 5.0 m. So at the widest opening set A preserves the local density and underpredicts the speed, set B improves the speed and underpredicts the density, and both underpredict the flow. That is a milder version of what Liao et al. (2014) saw in FDS+Evac.

Deviation of the simulation from the experiment, mean over six seeds, with the band at 6 % for flow and 10 % for density and speed:

The seed-to-seed standard deviation is 2 to 4 % for flow and density and 3 to 6 % for speed, so the differences between A and B at the wide openings are larger than the simulation noise. The experiment, however, is one run per width, and we have not quantified its measurement error; the band is an assumption and was not measured.

The honest summary of Step 5: a substantial improvement over the defaults and a partial transfer to the held-out widths. By the stated rule, the model is not validated for the full 2.4 to 5.0 m range. At the calibration widths 2.4 and 3.6 m both sets meet the tolerance on all three observables, but that is only a fit. The held-out evidence is two widths: at 4.4 m set B meets the tolerance on all three observables and set A fails it on speed; at 3.0 m both fail it on flow. And both miss the widest calibration opening in a way the two calibrations move between speed and density but neither removes. Whether that is acceptable depends on the purpose, which is why the purpose has to be stated first.

Step 6 — the same chain on the other models

Everything above is one model. JuPedSim 1.4.2 has six others. One is the original Collision Free Speed model, which is identical to the V2 used above when all agents share the same parameters. We ran the chain on four of the rest: the Collision Free Speed model V3, the Anticipation Velocity model (Xu et al. 2021), the Social Force model (Helbing et al. 2000) and the WarpDriver model (Wolinski et al. 2016). The fifth, the Generalized Centrifugal Force model (Chraibi et al. 2010), we tested without calibrating it. The chain needed little new code: a table of models and their parameter names selects the model, and screening, calibration and validation use the same Dakota files with a different variables block.

The comparison is less even than it looks. In each case the desired speed was fixed at the measured 1.55 m/s and every remaining parameter was free, over ranges we chose around the defaults. The Collision Free Speed model was calibrated in a box narrowed by its screening, from two optimizer starts. The other models got wider boxes, one start each, and 51 to 69 evaluations for eight to ten free parameters. A worse fit below can therefore mean a worse model or an optimizer that stopped too early.

Three models did not calibrate. In our setup the Social Force model could not be run at the measured free speed. With 350 people at three per square metre pressing against the boards, it pushed the front row through the wall within two seconds, and JuPedSim aborted the run. This happened at every radius from 0.15 to 0.25 m, at softer and harder body forces and at 1.4 m/s, and a smaller time step, down to a millisecond, does not help. The model runs at the default 1.2 m/s. With the desired speed fixed at the measured value, every evaluation would have failed, so we did not run the calibration.

The Generalized Centrifugal Force model fails in the same way at the 0.01 s time step used for all models: within seconds the front row is pushed into the boards. At a ten times smaller time step it runs, but the full chain would take about two days on our machine, and we have not run it.

The WarpDriver model ran everywhere, 180 screening evaluations without a single failure, and that is the problem. Averaged over the observables, only the radius and the desired speed have any effect. The six parameters that describe the collision prediction stay below a tenth of the strongest effect; on single observables two of them reach 0.11 and 0.14. The optimizer drove the radius to the top of its range and stopped with the flow 26 to 72 % too high, the density about 40 % too low and the speed up to two and a half times the measured one. No jam forms anywhere in the parameter box, and the crowd passes the gap at close to its free speed. We suspect this is because the model has no body contact, but we have not tested that.

Two models calibrated. The Anticipation Velocity model, with eight free parameters, converged in 51 evaluations to a residual norm on the six-seed means of 2.5, between the two Collision Free Speed sets at 2.4 and 2.7. It fails the two flow comparisons that both of those sets fail: the held-out 3.0 m flow, by 6 %, and the 5.0 m flow, by 11 %. Its radius came out at 0.137 m and its time gap at 0.65 s, close to set A. Its anticipation time and reaction time both ended at the lower bound of 0.1 s. The fit switched the anticipation off as far as we let it, so this calibration hardly tests what sets the model apart from the Collision Free Speed model. A wider box would have to be tried before reading this as a statement about anticipation in a jam; it may only say that this experiment does not exercise it.

The Collision Free Speed model V3 replaces the isotropic neighbor repulsion by an anisotropic perception field and a turning rule. With ten free parameters, it converged in 69 evaluations to a norm of 5.4. It gets the flow within 2 % at 2.4 and 3.0 m and then falls short, by 18 % at 5.0 m. Its density is 8 to 22 % too high and its speed 15 to 26 % too low at every width, so it packs the jam denser and slower than the experiment even where the flow is right. This is the right flow for the wrong reasons, as with FDS+Evac in Liao et al. (2014), but in the opposite direction: there the density was too low and the speed too high. The four new parameters are not idle. In the screening, the lateral range of the perception field and the body buffer rank right behind radius and time gap. With ten parameters, one start and 69 evaluations, the worse fit may reflect the optimizer budget as much as the model, so we read its rank as provisional.

Deviation from the experiment, mean over six seeds, with the band at 6 % for flow and 10 % for density and speed. Not every run got everyone through the gap within the time limit (28 of 30 for the Anticipation Velocity model, 27 of 30 for V3); all runs enter the means:

Two findings carry over from the single-model result. First, the 5.0 m flow is low for every model that forms a jam, by 10 to 18 %, so that misfit is not the Collision Free Speed model’s alone. Either the widest run differs from the others in a way none of the models capture, or the common scenario does; the synthetic lattice start is the obvious suspect. Second, a calibrated set is still one sample. We ran one optimizer start per model, and the two Collision Free Speed sets already showed how far apart two draws can be. Some differences are large enough to survive these caveats: the WarpDriver model forms no jam in its box, and the Social Force model could not be run in this setup. The order of the models that calibrated is less firm. The Anticipation Velocity model matches the Collision Free Speed model with its anticipation switched off, and the V3 model may do better with a narrower box or more evaluations.

What part 1 establishes, and what it does not

• A reproducible calibration. Every step is a Dakota input file and a driver script in the repository. Anyone can rerun it with another observable, model or experiment from the same archive.
• The defaults are not usable for this purpose. They miss the flow by a factor of two to three, and no single parameter fixes it.
• Two parameter sets. Two optimizer starts give sets with near-equal residual norms that differ by factors of two to four. The data do not identify the parameters; they constrain combinations of them.
• The chain carries over to other models. Rerun on four other JuPedSim models with little new code, it separates them: the Anticipation Velocity model calibrates as well as the Collision Free Speed model, with its anticipation switched off, and fails the same two flow comparisons; the V3 variant fits worse with a dense, slow jam, a provisional rank given its optimizer budget; the WarpDriver model forms no jam in its parameter box; the Social Force model could not be run at the measured free speed; the Generalized Centrifugal Force model needs a ten times smaller time step and was not calibrated. The 5.0 m flow is low for every model that forms a jam.
• Partial width transfer. Held-out widths inside one experiment are an interpolation test. The model passes it in part. This is not evidence for any other geometry, crowd or motivation.
• A pipeline audit. Between the first draft and this one, reviews found participants missing from simulations, unwritten final trajectory frames and a geometry error. Each was corrected and every dependent result recomputed; the Hermes steady-phase observables changed by less than half a percent. The record is the pipeline corrections section of the repository README, pinned to the commit this note was built from.

The natural next question is what the two sets do outside this experiment. Part 2 takes both to a half-metre gate and to an unguided entrance. One of them stalls, the other is too fast, and a second calibration on the new data disagrees with the first about which parameters matter.

References

• Chraibi, M., Seyfried, A., Schadschneider, A. (2010). Generalized centrifugal-force model for pedestrian dynamics. Physical Review E 82, 046111. doi:10.1103/PhysRevE.82.046111.
• Helbing, D., Farkas, I., Vicsek, T. (2000). Simulating dynamical features of escape panic. Nature 407, 487–490. doi:10.1038/35035023.
• Jones, D. R., Schonlau, M., Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13, 455–492. doi:10.1023/A:1008306431147.
• Kretz, T., Eckes, L., Knappik, L., Diz, J., Lipp, D., Müller, S. (2026). Using empirical travel time distributions for calibration of a model of pedestrian dynamics. EURO Journal on Transportation and Logistics 15, 100179. doi:10.1016/j.ejtl.2026.100179.
• Kurtc, V., Chraibi, M., Tordeux, A. (2018). Automated quality assessment of space-continuous models for pedestrian dynamics. arXiv:1809.01862.
• Liao, W., Chraibi, M., Seyfried, A., Zhang, J., Zheng, X., Zhao, Y. (2014). Validation of FDS+Evac for pedestrian simulations in wide bottlenecks. IEEE ITSC 2014, 554–559. doi:10.1109/ITSC.2014.6957748.
• Liao, W., Zhang, J., Zheng, X., Zhao, Y. (2017). A generalized validation procedure for pedestrian models. Simulation Modelling Practice and Theory 77, 20–31. doi:10.1016/j.simpat.2017.05.002.
• Ronchi, E., Kuligowski, E. D., Reneke, P. A., Peacock, R. D., Nilsson, D. (2013). The process of verification and validation of building fire evacuation models. NIST Technical Note 1822.
• Sargent, R. G. (1984). Simulation model validation. In: Simulation and Model-Based Methodologies: An Integrative View, Springer, 537–555.
• Sargent, R. G. (2008). Verification and validation of simulation models. Proceedings of the Winter Simulation Conference, 157–169.
• Seyfried, A., Schadschneider, A. (2008). Fundamental diagram and validation of crowd models. ACRI 2008, LNCS 5191, 563–566.
• Seyfried, A., Passon, O., Steffen, B., Boltes, M., Rupprecht, T., Klingsch, W. (2009). New insights into pedestrian flow through bottlenecks. Transportation Science 43, 395–406.
• Tordeux, A., Chraibi, M., Seyfried, A. (2016). Collision-free speed model for pedestrian dynamics. Traffic and Granular Flow ‘15, 225–232.
• Wolinski, D., Lin, M. C., Pettré, J. (2016). WarpDriver: context-aware probabilistic motion prediction for crowd simulation. ACM Transactions on Graphics 35(6), 164. doi:10.1145/2980179.2982442.
• Xu, Q., Chraibi, M., Seyfried, A. (2021). Anticipation in a velocity-based model for pedestrian dynamics. Transportation Research Part C 133, 103464. doi:10.1016/j.trc.2021.103464.
• Experiment data: Hermes bottleneck experiment, Düsseldorf 2009, doi:10.34735/ped.2009.6. Dakota: Adams et al., Sandia National Laboratories, version 6.24.

Code

Driver scripts, Dakota input files, results and figures for every step: github.com/PedestrianDynamics/jupedsim-dakota-calibration

Software used in this study:

• Dakota 6.24
• JuPedSim 1.4.2
• PedPy 1.4.0

By: Mohcine Chraibi

Drafted with an AI assistant; analysis, code and editing by the author.