Weather Underground GET request needs a timeout

[AdySan]

I think this is where the WeatherStation gets stuck sometimes. It just keeps waiting forever. I guess it would be a good idea to add a timeout here.

https://github.com/squix78/esp8266-weather-station/blob/master/WundergroundClient.cpp

 // This will send the request to the server
  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: api.wunderground.com\r\n" +
               "Connection: close\r\n\r\n");
  while(!client.available()) {
    delay(1000);
  }

[Wolfi2000]

I think it schould be done at all internet requests, not only while wunderground update.

[totof60]

Ander L add a soluce for that , and it work for me

Ander L say :
Hi,
@wolfgang : could try something like this:

// This will send the request to the server
client.print(String("GET ") + url + " HTTP/1.1\r\n" +
"Host: api.wunderground.com\r\n" +
"Connection: close\r\n\r\n");
int cntr=0;
while(!client.available()&& cntr < 3) {
cntr++;// Make 3 tries before quiting
delay(1000);
}

I've used something similar in other code for making sure the wifi is connected.

[AdySan]

Cool, thanks. Just deployed this on my WeatherBug, will report back how it goes. It generally seems to fail at night for me, works fine all day.

[stex2005]

@AdySan have you solved the problem by adding the timeout?

I'm using the stuff concerning the Weather Station found here on GitHub. I'm improving the project by means of adding a MQTT communication to the EasyIoT cloud server in order to remotely control a Thermostat.
The MQTT part works fine, but sometimes (expecially during night) I'm having troubles with the OLED, which freezes and stops working (during the update to wunderground) until a RST.
I can't understand where's the problem.
I've done some checks with WL_CONNECTED before requesting to wunderground, I've even added a timeout as suggested but it still freezes.

For a while I thought to add a softWDT to avoid this problem, in fact as I reset the node, the system restarts without any problems. Any idea?

---

WifiConnection code in loop:

if (WiFi.status() == WL_CONNECTED) {

if (readyForThermostatUpdate && ui.getUiState().frameState == FIXED) {
updateThermostat();
}

if (readyForWeatherUpdate && ui.getUiState().frameState == FIXED) {
updateData(&display);
}

}
else {
Serial.println("Disconnected.");
WiFiConnect();
}

void WiFiConnect() {
Serial.println("***************");
Serial.print("Connecting to ");
Serial.println(AP_SSID);
WiFi.begin(AP_SSID, AP_PASSWORD);
int counter = 0;
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
display.clear();
display.drawString(64, 10, "Connecting to WiFi");
display.drawXbm(46, 30, 8, 8, counter % 3 == 0 ? activeSymbole : inactiveSymbole);
display.drawXbm(60, 30, 8, 8, counter % 3 == 1 ? activeSymbole : inactiveSymbole);
display.drawXbm(74, 30, 8, 8, counter % 3 == 2 ? activeSymbole : inactiveSymbole);
display.display();
counter++;

}

Serial.println("");
Serial.println("WiFi connected");
Serial.print("IP address: ");
Serial.println(WiFi.localIP());
}

---

Timeout code used:

This will send the request to the server
client.print(String("GET ") + url + " HTTP/1.1\r\n" +
"Host: api.wunderground.com\r\n" +
"Connection: close\r\n\r\n");
int cntr=0;
while(!client.available()&& cntr < 3) {
cntr++;// Make 3 tries before quiting
delay(1000);

[AdySan]

@stex2005 Thanks for the pointer. Unfortunately adding the timeout hasn't helped as you observed too. I never really considered that the OLED could be freezing. I don't thing it's a problem with the OLED library, since I have other projects using it which are running flawlessly. Perhaps something to do with the UI part of the library.

[kentaylor]

This is an ongoing problem that I experience regularly and is also reported in #36 . @stex2005 reports and @AdySan reports that adding a timeout doesn't fix it, which means a function called
in this loop is not returning.

I don't know the solution but can provide some data. I'm confident it doesn't occur if the internet connection is good and the server to which it connects is working correctly. The evidence is that I have one device on a flaky network at https://thingspeak.com/channels/126118 and it fails much more often than one on a good network at home. The device on the flaky network sometimes fails on the Updating time... screen which I haven't seen on the good network. Thingspeak is flaky and fails in different ways. Most of the time this shows up as a gap in the data for both device installations. On one occasion it has shown up in both devices failing at the Updating thingspeak... screen at the same time. I've recorded the connection times of one simultaneous freeze failure on both devices. The simultaneous identical failure on different networks must be associated with an incorrect response from the Thingspeak server that the ESP8266 can not handle.

Interestingly Thingspeak is called in 2 places in the code. One Thingspeak http request never gets stuck and is called 5 times more often than the other Thingspeak http request which sometimes gets stuck in this loop.

The possible failure modes in that loop are client.connected() always returns true, client.available() is always > 0, client.read() doesn't return or parser.parse(c) doesn't return. @AdySan reports that it can't be client.available() always being > 0 because limiting the loop count to 3 doesn't help. Limiting the client.connected() loop is worth trying and hasn't been tested.

connected() returns false if _client no longer exists otherwise it calls state() and available() returning true if state() == ESTABLISHED or available() is true.

The client.available() function maps to a call to the Espressif library at https://github.com/esp8266/Arduino/blob/633e48f3aec5f1c3c11d4498fc90d378d49e6e9f/libraries/ESP8266WiFi/src/include/ClientContext.h#L208 which returns 0 if the socket is closed. Where does the value come from?

Next client.available() is called which calls _client->getSize(). _client is defined to be of type ClientContext*. ClientContext is defined and getSize() returns 0 if the the receive buffer is empty and the number of bytes buffered if it isn't. When the receive buffer is empty optimistic_yield is called which in turn calls ... Ultimately yield() is called which resets the watchdog timer so that is why it could be looping here without the watchdog timer ever timing out and forcing a reboot. To be continued.

[kentaylor]

Still getting stuck and able to observe that it was stuck here on one occasion.

Currently the weather station library uses the WiFiClient library for accessing web services. This requires that the http processing logic and request format be constructed in code. There is also a higher level HTTPClient library that could be used for these interactions.

[squix78]

Hi Ken. What is the situation for you on this issue? Did your fix solve it? I am running now my weather station for several weeks without need for manual restart...

[kentaylor]

I like the idea of using the HTTPClient library. This accesses web services at a higher level of abstraction than the WiFiClient library so for that reason alone I'd recommend the changes in the commit at kentaylor@f4bdc96.

Since having made that change I haven't seen a failure but I'm not sure if the HTTPClient library has fixed the problem. I previously had a device on a flaky network so I saw the problem often (every few days). That network is now fixed so I can't be sure if the code change helped. I have only seen the problem rarely (less than once per month) on a good network. I've also seen it when Thingspeak gets overloaded and Thingspeak responses become flaky but that was only once. If that happens, every weather station everywhere will fail at roughly the same time and have to be rebooted.

Edit: It occurs to me that it is not obvious why I am referring to Thingspeak when weather station does not use Thingspeak. The derivative of Weather station at https://github.com/kentaylor/esp8266-weather-station/ has additional DS18b20, DHT22 and BMP280 sensors that report measurements via Thingspeak.

[kentaylor]

The HTTPClient library has not fixed the problem. I now have a few devices on a Telco WiFi dongle and some have experienced the hanging problem again. I expect that is because TELCO networks are flaky at times.

This means that the problem is not in the Weather Station code and in particular is not related to the way the weather station library uses the WiFiClient library for accessing web services. The problem is further down the stack somewhere within the HTTP or TCP/IP handling.

I've seen it in other applications that use HTTP as well but never in applications that use UDP. If the device can not establish a HTTP connection it will be OK but if it establishes the connection it becomes susceptible to hanging when connectivity is lost part way through the transmission. It will continue to wait for data that the server will never send, even after network connectivity is reestablished. With UDP it doesn't wait which is probably why UDP doesn't cause hanging.