SUST custom feeds #6

New Issue

lucidiot · 2020-03-24T07:52:38Z

lucidiot commented

2020-03-24 07:52:38 +00:00

Four languages (english, french, italian, german) and two types: aviation and rail/marine.

The search form performs a POST request to the same URL (/index.php?id=...). Some URLs have simpler slugified versions, but the form still posts to the URL with IDs.

Four languages (english, french, italian, german) and two types: aviation and rail/marine. Example: [French aviation search form](https://www.sust.admin.ch/index.php?id=231&L=1) The search form performs a POST request to the same URL (`/index.php?id=...`). Some URLs have simpler slugified versions, but the form still posts to the URL with IDs.

lucidiot added the

feed

label 2020-07-26 13:52:56 +00:00

lucidiot commented

2021-05-22 18:27:44 +00:00

The site probably changed while I was busy implementing other feeds. To generate a feed, I would need to make a GET request to the form page, which gives me a cHash that I will need. I can then make a GET request which includes the cHash and other parameters, and I will get a table with 1860 hidden <tr> fields with data- attributes that include a URL to get a JSON payload for each of the reports. That makes for 1862 requests to generate the feeds.

I could implement that, but I don't want to overload tilde.town once a day and maybe get us blocked due to what looks like a DoS attack. The other option is to write a script that isn't stateless, which is definitely not what the whole ITSB pipeline is currently designed for, and only fetch reports that were not already fetched.

The site probably changed while I was busy implementing other feeds. To generate a feed, I would need to make a GET request to the form page, which gives me a `cHash` that I will need. I can then make a GET request which includes the cHash and other parameters, and I will get a table with 1860 hidden `<tr>` fields with `data-` attributes that include a URL to get a JSON payload for each of the reports. That makes for 1862 requests to generate the feeds. I could implement that, but I don't want to overload tilde.town once a day and maybe get us blocked due to what looks like a DoS attack. The other option is to write a script that isn't stateless, which is definitely not what the whole ITSB pipeline is currently designed for, and only fetch reports that were not already fetched.

Sign in to join this conversation.