In a Step Function we’re able to wait until a Glue Job is complete before moving on to the next step by checking “Wait for task to complete”. Tragically, not so with Glue Crawlers 🙁
There are several ways to wait for a Crawler to complete. The easiest is to poll the state of the Crawler using a Glue:GetCrawler step. If the output is READY then move on, otherwise wait for a period and check again.
Remember to add the Output from the Glue:GetCrawler step. This allows us to check it in the Choice State.
In the Choice State test that the $testCrawlerState.Crawler.State is equal to the string constant READY.