Explore the concepts of estimation, calibration, and validation in travel modelling workshops, emphasizing the importance of accurate model representation and parameter fitting. Learn from expert insights on machine learning, classical statistics, and the significance of model validation for replicating real-world scenarios.
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Estimation, Calibration, Validation: What, Why, How? Mausam Duggal | WSP Canada Travel Modelling Group Workshop | University of Toronto May 6th, 2020
Aspirations for all models (ABM and trip) hopefully Every theory, , distorts reality in that it oversimplifies. But if it is a good theory, what is omitted is greatly outweighed by the illumination and understanding that is thrown over the diverse empirical data. With four parameters I can fit an elephant, with five I can make him wiggle his trunk" Simplicity & sensitivity over complexity Avoid overfitting John Von Neumann (founding figure in computing) Model architect s triangle Paul Samuelson (father of modern economics) Evidence based If it disagrees with experiment, it s wrong. In that simple statement is the key to science. Richard Feynman (ten greatest physicists of all time)
What? Source: Rick Donnelly School of Travel Demand Modelling, Continuing Education Series Source: Travel Demand Model Validation and Reasonableness Checking Manual, 2nd edition, 2010. Cambridge Systematics
Estimation: How? Machine Learning Courses at University of Toronto: Prof. Eric Miller Prof. Narul Habib Classical Statistics TRESO CVS GPS GGHM GTAv4 York TRESO Mississauga Brampton Halton Durham Mississauga Halton
Calibration: How? Some consider it more art than science Fair to say that it is mainly science, with some art in the form of thumb rules The whole is more than the sum of its parts? Overall model should be able to reasonably explain travel behavior across key socioeconomic and geographic segments
Validation: How? Remember we are always modelling an average point estimate there is no variance in our parameter estimates unlike in the real world where preferences vary Are the traffic and transit counts average point estimates? In 90% of the cases, there was only a single day of count available and none of the locations were counted at the same time to preserve the law of conservation , where i = incoming links, j = outgoing links, and v = volume ?? = ?v Is it fair to ask a model that is a simplified representation of reality to replicate the Signal + Noise? If we cannot isolate the noise from the signal, we should assume a certain variance by link type or volume group or line ridership. The model must be validated to the variance Has the model really captured all the flows on the network? What about: o Uber, taxi, etc o Long distance passenger movement o Undercount in the TTS o Trucks?
Validation: How? Different applications of the model will require different levels of validation Corridor level analysis regularly uses demand adjustment for input in to simulation exercises A model being used to evaluate policy futures needs to get overall flows correct by broad road classes and screenlines Transit service level planning Etc A number of statistical measures available to measure a model s validation The ones routinely used: o Pctdev: percent deviation o RMSE: root mean square error o PRMSE: percent root mean square error o GEH: Geoffrey E. Havers o RNSE: root normalized squared error Testing for a model s sensitivity to key variables like cost, land use types etc. is critical Extensive literature on appropriate elasticities Compare results to other areas in the Region for a reference case
Application: How? possible plausible probable Single point forecast: Old way Range of forecasts: New way
So the real question and challenge is? Surely You re Joking, Mr. Feynman! If it disagrees with experiment, it s wrong. In that simple statement is the key to science. How will we forecast alternative futures whose behavior we can t see today but are routinely asked to do? Richard Feynman (ten greatest physicists of all time)
