How To Wait Until A Glue Crawler Is Complete In A Step Function

In a Step Function we’re able to wait until a Glue Job is complete before moving on to the next step by checking “Wait for task to complete”. Tragically, not so with Glue Crawlers 🙁

There are several ways to wait for a Crawler to complete. The easiest is to poll the state of the Crawler using a Glue:GetCrawler step. If the output is READY then move on, otherwise wait for a period and check again.

Remember to add the Output from the Glue:GetCrawler step. This allows us to check it in the Choice State.

In the Choice State test that the $testCrawlerState.Crawler.State is equal to the string constant READY.


This entry was posted in Programming and tagged , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *