Evaluation and the Smart City Challenge
[Please stand by for realtime captions.] >> [Please stand by for realtime captions.] >> Ladies and gentlemen thank you for standing by. The conference will begin momentarily. >> Ladies and gentlemen thank you for standing by today's program will be starting momentarily. Please continue to hold and thank you for your patience. >> Ladies and gentlemen, thank you for standing by. Welcome to today's evaluation and smart city challenge. At this time all participants are in a listen only mode. If you should require assistance during today's call please press star and then zero. I will not turn things over to Brian Cronin.
Good afternoon everyone, thank you and welcome to the The Smart City Challenge evaluation webinar. And I think our last webinar and then we heard you loud and clear and we will let you go write your proposal we felt like and you felt like the opening kickoff meeting a few weeks back that we should die a little bit more into the evaluation component. Today I am going to start this out and we will kind of go through the Notice of Funding Opportunity to make sure it's clear as to what we are expecting you to tell us as you submit your proposal of that were going to talk about our expectations related to performance measurement and evaluation and talk a little bit more and answer your questions and then Kevin will provide information on other technical resources and information that out there that should you need more information, you're obviously welcome to call at but we wanted to highlight those resources and there is some next steps information and we will talk about that. Thank you and as a reminder, we start looking at evaluation, we are talking about the outcome for the smart city challenge. As he made clear in the notice is really all about improving safety, mobility, and opportunity and arresting climate change. We are looking to capture measures and determine the outcome from the The Smart City Challenge on these different challenge area.
So in the goals on page 10 and 11 we talk about really starting at this is exactly your challenge so while we put a framework around evaluation talk about different performance measures it is ultimately grounded in the challenges you face. So Vulcan Inc. understand the impact from the technology, the applications, that will help address these challenges. Looking at evolutionary steps to integrating technology so were looking at how to demonstrate and evaluate the impact of the technologies also understand the technical policy and institutional arrangement needed to make this change. We are want to look at reproducibility so we really want there is an aspect of the evaluation to try to understand how this will work not only in your city of other cities across the country and tie into our ability to do technology transfer and outreach without thinking to go through the development process. And then ultimately we want to work with the federal partners and look at how we accelerate the deployment of clean transportation and connected and automated vehicle technologies.
In the notice in section A we talk about performance measurement plan and about U.S. DOT tracking surveys that will be going on during the time of this notice. That we will talk about data management and support filters article roles in how you do anything of management plan how to support independent evaluation, how you write data to support that evaluation and providing some data on it research data exchange so others can analyze and look at the research data to help understand how does technology is impacting their cities. And what you're going Dto need to do in section talks about the application process and the technical approach around doing those things. The performance measurement plan looks at identifying performance measures and quantitative performance targets. What we need to discuss this type of data needed to support performance measurement in your demonstration and evaluation. We want to look at hypothesis and methodologies for collecting data before hand, data during the lifecycle of the demonstration and as well as cost data and information on the timeline and application availability deployed in your demonstration. And how the performance recognition will be released as open data and data for other researchers and technology we move forward.
In the notice we also talk about setting different performance measures so for instance some items we identified as possible was that would make sense in a smart city application around traffic related fatalities and injuries, transition to clean multimodal this objective, cost savings, and you can read these but it's critical to identify your performance measures how you want to do that in the smart city. How you do your data management and support for independent evaluation. It going to be we are providing independent evaluation undergirding the data that you generate. They are going to look at the hypothesis that you recommend and work with you as you understand which the greater going to collect and how they're going to be able to assess if you have those goals so they will be looking at on the data that qualitative evaluation methodologies to look at before and after, cost-benefit, user acceptance or satisfaction, document lessons learned, and your purchase in overcoming challenges as he went through implementation, costs and impact, and really when it comes down to these you achieve your vision? Was the demonstration ultimately achieving the vision in the smart city challenge.
Continuing more on the data management and evaluation report, unique evaluation support plan how you are going to work with you and -- independent evaluators. To get clarity as we get into the demonstration of roles and responsibilities and is really about understanding what the hell you're going to collect, and the metadata to describe that data as to what the data is in the purpose and the how and what it's for. How you're going to monitor the performance measures and target, availability talk about how the site and the staff are able to help the independent evaluator in terms of their role in providing that data and information to them so they can understand your data and applications from that. If we decide after a and interviews are needed, to supplement evaluation then helping to do that. Your role is to support the independent evaluation, access to the site, in fact staff to conduct experiments and injuries.
Considering on your systems must be capable of generating the data needed to calculate measures that you well-defined so we need to be looking into how well those systems are performing and what's needed to generate the data you need for your evaluation and measurement. You have to look at monitor the quality of the data if you go mom and mom were you thought we were collecting data and then you shouldn't have any copies have processes in place to be able to really understand your getting the data needed at the time of its and accurate and quality for supporting. And then transmission of that data to support the evaluator. Data collected for the evaluation will be owned by the U.S. DOT and required to support independent evaluation must be shared.
We have this research data exchange and that's only been using to share research data with the community at large. We had various connected vehicle data samples, we had data from the safety pilot, we had data from the connected vehicle pilot, is not necessarily your whole data set from the smart city but indicated that at least representative enough that can be used and we would like to house some of the research data exchange you will is articulate to us as we go through and part of the data management what information is going to be available and shared so we can make them available to the research community. That for the want to analyze how a smart city and technologies and applications can be used to support their city.
In section D your proposal should really provide measurable goals and objectives describe your approach for monetary impact on the demonstration across major different goal areas and then identify a set of target for measures that relates to the impact of your proposed demonstration. With that foundation I'm going to hand it over to Marcia Gulati for more details about independent evaluation.
Good morning and good afternoon to everyone who is joining us on the phone today. Thank you for walking us through the elements of the tran08's everybody is clear on the requirements that are expected related to evaluation and at the end of the short webinar if you have any questions, please make sure to ask us for clarification so we know that you know that always requirements are understood the way they are listed. For the next few minutes you will should focus and I will be talking to you about the more practical considerations as you develop your performance measurement plan and the other documents that Brian described. >> What you are about to do is truly the big and bold and exciting. And it is indeed a once-in-a-lifetime opportunity to be a living laboratory for the nation with respect to smart and connected cities. They not only demonstrate the innovative use of technologies and policies and partnerships but you realize you your role in ensuring the nation and probably the world because it will be international interest, the amazing things that you have accomplished with investment that is being made here. So you have truly some unprecedented opportunities to baseline where you are and then show the improvements in your city with the use of these new technologies and approaches and partnerships. You have even more exciting opportunities to develop compelling performance measures and innovative ways of testing and evaluating them. Your efforts will serve as a map for other cities and also an example for the people in United States to provide them very quantifiable and dependable benefits related to this you chessmen that we are making.
Given this, there is or could be our example of our thinking about logical way to approach some of the questions you should probably be asking yourself as you develop the elements of the NOFO related to evaluation. These are some fairly big strategic lessons. Several questions that may drive many others is in the end what do you and your partners really want to get from this investment? Not only investment of federal money by the investment of resources in your jurisdiction and your time and your people and your brainpower. What do you want to show the world and tell the world immediate about what you have accomplished with the The Smart City Challenge. And what you want to have learned from this grand adventure. For each jurisdiction the answer to these patients will definitely be different but ultimately these are the questions I think that you need to start thinking about. And then this starts to drill down deeper from these larger questions actually, the performance measures and hypotheses that mean the most to you. Once you are clear on those, it's kind of like reverse engineering, you need to start thinking about what data you will need to test those hypotheses and those performance measures. What data we need from who and how much? How long is it going to take to get it? What kind of agreement you have to have a place in order to have those data? And I make a point of this because it's not just transportation data, that you will want to be dealing with, it's data from all sectors of the city because the smart city challenge isn't just a transportation project. This always to be very clear that in the near term is also important in terms of baselining. You will need to give baselining some serious thought because it needs to be very clear on what data you need to do that because you don't really get a second chance to do that baselining and at the end of your project one of the most people questions or things that we tend to ask ourselves is we really should have collected data on X because that means a lot for stakeholders and ourselves. Think in advance about your baselining is important.
I get some guidelines in terms of approach. Set your performance measures that quantitative performance targets with your stakeholders at the beginning of the demo and articulate that in the performance measurement plan. You are going to want to think about the most effective way of partnering with independent evaluators and articulate that in your proposal. We like to see you establishing hypotheses and here is an example. Hypotheses are typically opening statements if I do ask them why will happen. If transit signal priority is deployed on all city buses, then 80% of all buses will arrive at their destination on time. The last one is establishing baseline conditions so you can establish a before and after jurisdiction.
As mentioned before a good evaluation allows you to highlight the amazing things you have accomplished and show off potential of investment for that technology. Another point of values that you gain visibility into the effectiveness and operation of all aspects of your city that you currently don't have on a very granular way anyhow an opportunity to go cross sector partnerships that are in some ways based on the kind of data exchange you set at the beginning. And keeping in mind that this is not just a transport project. It's critical to start thinking of what kind of data do we need to start thinking about between the transport and the healthcare sector, or public safety and integrate those into the performance measures that you will develop particularly important in addressing the outcome and the NOFO related to letters opportunity which is not specifically directed to just transportation. It's about economic development and society and jobs and the economy so you need to begin to be able to answer questions for city officials other decision-makers and your citizens about the performance of this project and not only at the end.
To that end, the evaluation itself cancer multiple goals. To replicate -- not only identified what is going right what's been done well, but to also identified what is not going well and why. Hopefully most things you put out there in the environment will go right but some might not. Honestly, much of this challenges so on the cutting edge of so many things that you will be surprised that everything you thought would go right actually did but it's important you document even those instances where things didn't go as well so other people can learn how to mitigate those problems or avoid them. Because you will be doing something that no one has tried before. You will invent new ways to solve new problems in knowing what you try, will reduce the time, that frustration ultimately for you and your colleagues and also for those of you who are representing cities that may want to pursue this in the future.
We also ask you if it helpful as a navigation tool imagine you were the second city trying this The Smart City Challenge . What would you want to learn from the first in order so you can be successful? Treat that city the way you want to be treated. Try to get the details of what we expect to happen, and if it doesn't, happen what does that mean?
Some more technical guidance on best practices. It's important to choose performance measures that are of value to your stakeholders is important to talk to them about what's important to them. It seems to be a no-brainer but sometimes these discussions are left off the table. You need to establish clear traceability for goals to measures and from system input to outcome, this will help guide your data collection and also your hypothesis development. You need to develop a data management prospect is clean and efficient to integrate multisource data into a common format and database. This is only help you and the need to have a plan for automated to verify your data quality and perform periodically to ensure that whatever equipment that is being used to generate the data you need is accurate and in a healthy state.
This is just a summary of what we have been talking about. There are three major elements here the first one the The Smart City Challenge the bullets underneath are some of the things that you will be expected to provide us information on and then feed those and work with the independent evaluators who undertake if not all at least the vast majority of the bullets listed under independent evaluator. And also to be able to figure out what is going to be placed in the RDE so other jurisdictions can take advantage of the hard work that you have done.
Just so you know that all this work is done in the context with JPO in a larger program that essentially starts to track all over the United States the performance of various ITS systems the effectiveness of them, the installation and eventual replacement of different generations of ITS technology and it supports also making strategic effectiveness assessment about what gets deployed, the weight of employment, dealing with the development across the spectrum of ITS so just know that the results from the smart city challenge evaluation will be part of the larger scheme of things and that goes back to the notion of being a living laboratory that can help the Department of Transportation actually get a granular view into how this next generation of ITS takes off and is deployed.
We thought we were going to hold for questions until the end? There is a couple.
I will read the first one. Do we need to develop an evaluation support plan as part of the proposal, or do you just want to see evidence that we will be able to create a strong plan?
We've had this question for the other called we don't develop a plan in the proposal to provide evidence that you have the capabilities to do so.
Brian?
You don't have to create the evaluation support plan as the proposal, you need to show evidence that what to do and then we are hiring the independent evaluator but you need to have support to see -- feed your data help create your evaluation plan and support of that.
The bottom line U.S. DOT will contract separately with an independent evaluator.
I don't see any other questions in the chatbox. Kevin, do you want to?
Just add the questions and we will come back to them. >> Technical resources now we are for weeks and 30 minutes from when the proposal is to do so you have seen much of this information before, this is just a reminder that there are resources out there and if there's something missing here, the other things you are addressing in your proposal, that
-- you can ask up, we can research it. This list of eight information sessions today's webinar will be number nine, that will be up in a day or two. This'll be available also. Again most of this sessions were from December and January not only the presentations are up on the side of the transcript as well as you can replay the whole webinar obviously most don't have time to deal with that at this point, but it's available.
Next I know that were never questions related to system engineering. Federal highway begins a site and decide training the consortium for ICS training and education. The federal highway does have some good not only training but material on how to do a system engineering plan so this site is a good resource and again working with the visual audio team that you're going to have to point to will be able to prepare and of the system engineering work later. I would hope at this point everyone is familiar with this.
And again on each presentation we have the URL, you will have to cut and paste them.
The next few slides are from the ITS JPO sites so this is just to cover lots of materials, more than you need for addressing your technical proposal. We just provided overview here and this is again, most of you have seen this. So what we want to highlight here are connected vehicle courses -- we just had a little technical issue. This gives an overview again and I think we have presented as material and our previous workshop. >> If there is one site that all of a lot of use for any of the smart city is this is what we have for connected vehicle pilots because in part this is in some respects similar to [Indiscernible] some of the connected vehicle aspect probably not to the same skilled than you do so much more. There are ways of dealing with the performance measurement, dealing with security credential managing system, and then we have presentations, reports, guided material, all in this site. This is a very rich resource and will continue to be of use to all the cities going forward that you're looking to do this type of work or as part of future projects.
Similarly we have much more detail on ITS national architecture that the next layer down talking about connected vehicles is really another addition to another layer on the ITS architecture. For the most part a smart city in highly connected so much of what's applicable in the connected vehicle reference interpretation architecture will need to be part of your project for your technical proposal, to the extent it's a matter of how much in-depth you're going into this area, this is certainly a resource that should be able to address anything you need at this point. Again we can provide the experts in this area and provide responses through [Indiscernible] if there's any questions that come in the next few weeks. The last item supported from the ITS site since this is a key aspect of what we are looking and smart city is accessible transportation. There is a multiagency project that we host that involves folks from DOT and outside calls the accessible transportation research -- accessible transportation technology research. A lot of material on characteristics, technology solutions, not just pure transportation but ways for dealing with transportation, units of transportation, dealing with for a range of disability categories. And that is the last resource related question. The one topic we want to hit next before questions is general next steps because [Indiscernible name] is here.
Hey guys, I hope everybody is doing well and just wanted to talk a little bit about the upcoming city benefits before I do that I want to highlight what candidates talking about in terms of technical assistance resources. There might be people in your city there somebody that in your city that's just really focused on connected vehicles or somebody might be outside of your team, this is an opportunity for you to connect folks across your network to these various resources and communities of practice. So this is less about getting your information for your proposal and more about helping you get list over all on terms of the smartest of your city. If not limited to just the people writing your proposals I hope you will think about being the resource of resources for all those folks around your city and your partners. A couple of things on next steps for city business. I don't have anything on the slide but just some clarification. The secretary is coming, I think you all have your scheduled dates and times, it's a two-hour visit, the first 30 minutes will be that the mayor and the secretary. We are open to the mayor having another -- somebody asked the mayor to bring the governor, yes, we want to keep this a very small meeting we would encourage you not to open it up to your private partners, keep it governmental body can be just a one-on-one meeting the mayor and the secretary but we do need to see who is in the meeting. Let us know who's going to be there beyond the mayor. Two is the Roundtable, we will give you an invitation we have to see the list. I know some of you have unique circumstances we talked about about how we will configure that Roundtable but we still need to have a list from you for our counsel review. This is just a standard operating procedure we use that the secretary the. We ask that you submit that I think I sent a spreadsheet for you to put that into it if it's too hard to find, I will take it in an email, and is to be name, title, organization, and then once our counsel review that, we will give you the invite to share and manage all that and it will get a final RSVP list from you ahead of that. If anybody wants to talk off-line about your individual circumstances, I'm happy to do that. The third piece will be a press -- or of a gaggle pull aside, it's going to be open press but the press won't be allowed to ask any questions in that environment. So we will have a time set aside for press afterwards for the mayor and the secretary for any press questions but not a formal press conference style. In terms of driving press to your event, either done or Andy will be in touch with you folks in the mayor's office. Questions, comments, and love you guys are all sending me your -- I don't have too many list so we would be good to have that. So we can get those vetted.
Absolutely.
Thoughts about any of this? The city visits, evaluations, -- I have one more thing after we talk all this through and then remind me to come back.
Folks, the phone lines are open. If you would like to speak you may have to take yourself off mute?
Let's just check in.
Anyone from Pittsburgh on? >> Rob is here as well as the rest of my team.
Are you guys done with jet black?
No. I'm still jet black right now. That was very excellent, thank you for hosting that. I think we are on schedule here. We have to run our list of participants by the mayor and then we will get it to you as soon as possible.
Any questions on the evaluation?
No.
Just as a reminder folks, just tell me in your chatbox if you're going to send me your stuff. Kansas City, how are you guys doing?
San Francisco?
We are doing good. Where just working through the multiple [Indiscernible] we have heard from you guys that may eight was actually confirmed so we can start preparing a list.
When I go back to my desk, I will send you a new email.
Denver?
You guys doing okay?
I can't hear you. [Laughter].
Portland?
What happened to everyone? I know you guys are on.
I'm here, it's Catherine, we are doing good.
Can you guys send me your list?
We will have some more eyes 2.--
We will send it to you.
With that Portland?
Kansas City? Have we heard from Austin? Who am I missing?
Columbus.
Randy? Who is on?
I know you guys are out there.
Katie, are you still on?
Okay, send me your list to out. Real quick on June 7, 8 we are doing oral presentation on June 7 and eight, I think you got the amendments, you want to highlight was in the amendment?
In our amendment one we just tried to give a framework of the plans for orals, the two components being technical teams with the 45 minute presentation on the topics prescribed in the NOFO and the second component is the mayor's pitch .
45 minutes of presentation and time with the Q&A with the technical review panel. That presentation is just your team in the room with the technical review panel, not everyone listening to everyone else's oral presentations so it will be a 45 minute presentation, you have to take all of it, if you only want to present for 50 minutes, that's fine. And then open Q&A so they can get a clarification or more of a conversation. For 30 minutes after the 45 minutes. We anticipate having for presentations on the 7th and three on the 8th we get a schedule out. We will set up a camp for you guys so you have your individual rooms and you can Out in your future headquarters and then on the 8th there will be a public event where mayors will make a pitch that also include the video and the 3D representation. Autodesk or other. C gas can start thinking about that as well. I know you're still writing so you can't put it all together but as you thinking about schedule, the time on the 8thafternoon of the is TBD and we would like all the players to be here but if the mayor cannot be here, he's designee, and the location is also still on that TBD, it will be here in Washington but you are still exploring where it will take place. We will try to live stream those pitches and open them up to a live audience and open press as well. The 45 minutes and 30 minute presentation is just you, your team by herself any technical panel and the public event on the 8th.
The oral presentation part one just as a reminder you can have up to six people presenting, you have others in the room a maximum of six people presenting. For part two the mayor's page, we have indicated the mayor and up to two other personnel as well.
That a shorter pitch. The mayor's pitch is a 10 minute maximum to include a two minute video.
Any questions on that? The amendment went out on The Smart City Challenge so if you have not seen it, send us a note and we will resend it.
It was issued on the 18th and I sent it directly to all finalists. Location wise to keep in mind the technical presentations are anticipated to be in the media center here at U.S. DOT.
Parts to mayor's pitch is TBD but in the DC area.
When are you anticipating to give up a schedule of the city's?
We are working on it, I can't answer that question because I don't know the answer. We will try to get it to you as soon as we can.
Okay.
Other questions? I got excited? -- Are you guys excited
[Laughter].
There was one more question. What are the relationships between the elements and this challenge? >> The ITS architecture is a that interoperability for our information and data across ultimately all transportation systems. For the smart city were moving into some data elements that go beyond transportation so they may not fit exactly into the ITS architecture that you will need to have a plan for that. All the elements are addressed by the architecture but the transportation is you have a framework for all your information will be interoperable and how you will be able to share data. Anybody else from DOT [Indiscernible - too far from mic].
There's also the elements that part of this effort is also to be able to feed updates and enhancements to the architecture, because there are new element to the architecture and the CVRA that the smart city challenge will address in the course of business so will be great to get that input in terms of data flows.
Ultimately were developing modifying standards and work with industry to change status all the time that this will lead to an automated vehicle , there are very few interoperability standards.
One of the things they had anticipated when this idea putting glimmer and I is that to really be able to begin to thoroughly explore the data exchanges between the transportation sector and other sectors in the city's economy. To and from. The ability to get some visibility into does bidirectional data flows between transport and other is going to be invaluable going forward in terms of creating space and architecture for all that. The back help? -- Does that help? Phone lines are open.
She said it does.
Other questions, comments, clarifications?
Nothing?
Somebody is typing. We can hear you typing. You can just talk.
Thank you for your participation today and certainly continue to submit any questions you have to The Smart City Challenge
@dot.gov..
On the technical committees you expect the mayor to be in that technical committee or --
Is totally up to you. Right, the mayor and the technical committee is up to them. We would like for them to do the pitch. We hope that they would come from the public pitch and they are welcome, that's really your call whether you want to have them in the technical pitch.
Of the technical pitch is really do not thimbles so you're looking for --
Thank you.
Other questions? You guys know where to find out, don't hesitate to send questions, I will look for your lists, we will get the final schedule out to everyone so they can have it again for travel and we will talk to guys soon. Thank you.
The dots complete our conference for today. Thank you for your participation. You may now disconnect.
[Event concluded]