Yes! The NOTAM SPRINT is complete, and we achieved our goal!
After five very long days and nights, we have a working solution, that fixes the Big Problem. Not just an idea, not just a blueprint, but a working model that ingests all NOTAMS for a flight, and outputs a simple, colourful, pilot-friendly briefing the way we want it.
This post will (try to!) condense 120 intense hours of parallel group discussions, video calls, coding work, design, and progress into a readable synopsis. We’ll explain what we did, and how YOU can benefit from this: we’re doing this so everyone can see better NOTAMs.
This is how they are made, and we’ll get into the details further down …
Quick refresh: Us, and our goal ..
The Notam Alliance is a team of NOTAM end-users: pilots, dispatchers, airlines, and aircraft operators. We are spread out across the world, but we all agree: NOTAMS are awful, and we need to fix them. The Big Problem is this: PILOTS CAN’T SEE THE CRITICAL STUFF (read more on that).
For almost 60 years we’ve seen various attempts to try and improve NOTAMs, with improvement only beginning to be seen in the last few years. In 2021 we ran a Global Campaign with ICAO, IFALPA and IFAIMA to eliminate “Old NOTAMs”. In January 2023 the FAA system failure brought industry attention to the problem. In February we took on a challenging question: “What do pilots want?“. After 8 weeks of group effort (50 people took part), we came up with a list of 50 tags. In March we ran several weeks of tests using Large Language Models (LLM’s), like Chat GPT 4, to see how well random NOTAMs could be understood by a machine. The results show that Artificial Intelligence has reached a point that it can understand a NOTAM with >98% reliability. That means that instead of having to rely on source changes at the NOTAM Offices, we can use AI instead.
And so we ran a NOTAM SPRINT from May 8-12, 2023 to bring all of these developments together. Our goal: Design a prototype system to post-process NOTAMs, tag them, summarise them, and then sort/filter into a newly designed briefing package. It’s called a sprint because we aimed to do in five days what might otherwise take a year or more.
And the result: SUCCESS! Read on.
What happened in the SPRINT?
Monday was welcome day – introductions, setting out our goals, and forming teams. On Tuesday we got stuck into the work – focusing on five key areas to collaborate on: the NOTAM source, the list of NOTAM Tags, how we use AI, creating a Matrix to sort NOTAMs, and what the SuperBRIEF should look like. By Wednesday we had already seen much deep discussion on the finer points of NOTAMs. Thursday was a tough one: various roadblocks and hurdles to overcome, and little drop in energy, but by Friday morning we were seeing glimpses of a working model – and late in the day we decided that we’d use the Hawaii 🌴 timezone to give us all the hours that Friday could offer! Late on Friday night, we were able to tick off all five boxes in our checklist, and see a working prototype in action.
Over 300 people registered for the event, making this without doubt the largest collective group to address the NOTAM problem in history. We had a large contingent of commercial pilots (175 in all) – airline and non-scheduled, helping drive our direction along with dispatchers. Individual airlines showed up to help in good numbers: we had representation from United Airlines, Air Canada, Air New Zealand, Aer Lingus, UPS, Singapore Airlines – and many more. Flight Planning providers Lido/Lufthansa and Flightkeys were involved (thanks again for the supberb “Dark NOTAMS” concept!), as well as NOTAM Officers, IFALPA and IFAIMA representatives, and of course ICAO – who have been instrumental in NOTAM progress since we first started. And of course, everyone on the OPSGROUP Team showed up every day! Overall a fantastic mix of NOTAM expertise and user input to help shape the outcome.
SuperNOTAMS – a first look
OK, to the result: here is what we made. We’ll go through each step, and include the key points raised by the Sprint participants.
Some critical things to highlight before we get into the detail:
- We’ve built something that works. But there is a next step: getting people to use it. The heavy lifting it done – now we want to see it adopted beyond our core group of airlines and operators.
- Our target audience for wider implementation is the people that produce the NOTAM Briefings: Flight Planning Providers, Airlines, and larger aircraft operators. This is where the 50-page briefings come from, and this is where we can help fix things. Some providers and airlines are already starting to figure out how to integrate this into their systems, but we want to make it available to everyone.
- The blueprint and code is intended to be open source, we will share it with you if you need it! There isn’t yet a public-facing model for demo purposes, but we’re working on it!
- This is a middleware solution, not a front-end app. That means we’re not making something that works on your phone, we’re making something that sits in the airline OCC, or within the flight planning software. So, if you want to see this work at your airline or operation, let your IT folks know about it!
Step 1: NOTAMS
The source question was our first big discussion. What list of NOTAMs do we start with for a specific flight? There are various options.
- A list of NOTAM ids (EINN 0123/23, etc.) generated by the Flight Planning system for our flight.
- A list of ICAO codes parsed from the ATC Flight Plan, with the NOTAM messages obtained via DB query or API call.
- A freetext list of ICAO codes manually entered
For the SPRINT, we used this ATC FPL query box to get the relevant NOTAMs.
Step 2: AI tags NOTAMs
For this stage, there are some key elements.
- We start with a list of 51 different tags (like Runway Closed). We determined these in a working group of 50 pilots in early 2023.
- There is a tag to suit every type of NOTAM, so each one can be tagged with its’ “subject”.
- We use Artificial Intelligence at this step to apply the tag.
- This is done by using an API request to a Large Language Model (LLM) – we agreed on gpt-3.5-turbo from OpenAI after experimenting with other models.
- The API returns both the most logical tag for the NOTAM, and a 7 word summary in plain English.
- We had to extensivley refine the prompts used to get a reliable result ( >98%)
- Any tagging errors were usually only off by a “position of one”, for example an ILS U/S NOTAM might be tagged as Approach degraded instead of Approach not available.
Above is a sample of tags – showing four of the eight categories used.
Step 3: Rules Applied
For this stage, we used a Matrix to match different types of airports with the NOTAM tags that we want to see for each one.
- We used four tiers: Departure/Destination airports, their associated alternates, enroute alternates, and FIRs
- For each type of Airport/location, we selected the NOTAM tags we want to see first in the primary briefing.
- The Matrix is fully changeable by the airline or operator to match their needs.
Here, the group introduced the DARK NOTAMS philosophy – credit to Flightkeys for this one!
Very simply, the briefing is divided in two: A Primary section, and an Appendix.
The critical NOTAMs, and relevant operational ones, go into the first part. Everything else (DARK NOTAMS) – is sent to the Appendix. This allows the flight crew to quickly review the key NOTAMs, with the rest available in the Appendix. This idea was adopted by the group for use in the final briefing for the prototype, and works well.
Step 4: The NOTAM SuperBRIEF!
The final step was to wrap all the work above into the actual NOTAM Briefing. We called it the SuperBRIEF because it’s brief, and super! 😊
- We focused on creating a text-based PDF, as this is how NOTAMs are appended to our current briefings. A digital version with hyperlinks, and click-to-expand was also created, but outside the scope of our SPRINT objective.
- Graphical elements (maps, etc.) were also outside the scope.
- Different members of the group came up with a variety of suggestions. We incorporated these into a final draft to progress the prototype
- Below are some earlier designs from the FixingNotams.org team, and the final draft for the prototype.
The draft briefing above, used in our prototype model, shows how NOTAMs end up appearing in the Primary section of the briefing. They are divided by category (like Runway, ATC, etc.) and then by tag (like Runway closed). Colours are used to differentiate different categories of NOTAM. The most critical appear first. A pointer at the bottom lists NOTAMs that are in the Appendix.
And – that’s it!
This doesn’t fix everything! But it fixes the Big Problem.
It doesn’t solve it at the back end (the NOTAM Office) – but it doesn’t need to, anymore. Until this year, we were all convinced that the only way to fix NOTAMs was to change the way NOTAMs are issued by NOTAM Officers, perhaps adding a tag to each one so we could organize them better. But, AI has launched a cannonball of change into the aeronautical information sphere. NOTAMs are just the start of the change we can expect to see.
It doesn’t solve it only at the front end, either. This isn’t intended to be an App that just works on your iPhone, which might be handy but doesn’t fix the 100-page briefing package you’ll get in your EFB.
But DOES solve it in the middle. The model can work in any of these places:
- In the Airline OCC. The code can maintain a list of NOTAMs in a database, and fire off requests to the AI via API to tag new ones that come in. When it comes to running a briefing, just import the ICAO codes needed and a full SuperBrief can be generated.
- Within the flight planning software. The code can be part of the Flight Planning software, and be used when the NOTAM portion of the briefing is generated.
- At the NOTAM database holder. There aren’t many of these – EAD, the FAA, DINS – but it would be straightforward to include the code at the database holder side to tag new NOTAMs. That way, any flight planning provider can get a list of NOTAMs with their tags ready for sorting.
Iteration – the next step (+technical info for the geeks!)
Friday, May 12th – at 23:59: a working prototype is up and running. We plug in some NOTAMs, and we get the result we want – tagged, summarised, sorted, filtered. Now what?
We iterate. Keep working on the model to make it better and better. The hard part was proving the concept and making that first model. It works, and now we make it better. We’ve already done the first steps of iteration over the last seven days, working on the model to improve it well beyond the version we started with on Friday night. Deepest thanks to the handful of pilots and coders that plugged away to develop it further! Here’s a rough log of changes and improvements, to show we’re we are now at.
- NOTAM Batching. Our initial version of the protoype used sequential API requests, one for each NOTAM. It worked, but it was slow. A full briefing was taking 3 minutes. We changed to batching NOTAMs – working out the number of tokens we could send in one prompt, and then processing 5 or 10 NOTAMs in one request. That improved things, but it was still slow.
- Parallel API requests. We sent the batches above in sequence, but found significant speed increases when we changed to using almost parallel API requests, offset by 20 ms or so. Now batches of NOTAMs could be processed in parallel. We reduced the processing time for a full briefing down to as low as 20 seconds.
- Prompt splits. We found greatest reliabliity of results when we used split prompts to the AI model. We used three prompts. The first to send the list of tags. The second to inform the assisant of its role. The third to send the NOTAM batches. This was done using the “history” format of the API request model, meaning that in practice only one API request is sent but with three prompts.
More to come!
Thanks team!
At a rough estimate, our group of 300 particpants put over 1000 hours of effort into making this prototype over the five days of the SPRINT. That’s a really phenomenal gift from the “NOTAM community” to flight crew and dispatchers everywhere. Some went way above and beyond – working on the code late into the night, and iterating it further over the weekend. After 60 years, we have something that solves NOTAMs, and that’s absolutely amazing.
If you’re one of the 300, WELL DONE! Thanks for showing up, thanks for getting involved, thanks for the discussion, input, and great ideas, and thanks for designing, coding, and developing the idea.
Now for the final journey – seeing this implemented at scale. Let’s make it happen!
The NOTAM Team
team@notams.org
