Sensor stops sending data/crashes after x hours?

[ghoeffner]

Firmware : "NRZ-2018-123B" (eng) - might be related to #350 which I don't fully understand.
I just built this sensor and I noticed that after 1-2 days it will just stop working. Is there any other firmware I could try?

Thanks!

[brandungskieker]

Same here. The sensor even transfers corrupted data from time to time.

https://www.madavi.de/sensor/images/sensor-esp8266-830217-bme280-1-week.png

Because of many reports it might be a software issue rather than a hardware defect.

[dirkmueller]

@ghoeffner is this still an issue with newer firmware? I have never hit this problem so far..

[Tavas666]

Since 1/2 November I have several sensors that regularly stop working. They only restart after a hard reset. Could this be due to a new firmware update?

PS sensors are spread out over Flanders so I don't have access to check the firmware. I can only do remote hard resets (via a wifi switch).

[dirkmueller]

Yes, a large update was made available to OTA on October 31st.

Would be good to know something more about what the circumstances are, because this doesn't seem to affect everyone.

[RikDrabs]

That's about the same time as my problems started: on more than 6 sensors, that i know of, the WiFi signal spikes to +30 dBm on the Madavi graphics, at the same time the measurement data disappears from the graphics. I already reported this problem in issue #541 which was explained as error 31, and was (half) repaired by modifying Madavi graphics, but the measurements still stop temporarily. The WiFi ssignal only "looks" more stable. Since #541 seems to be linked to the problem reported by Tavas666, maybe #541 can be closed ?
The same sensors which produce these errors described in #541 (my sensor at home is one of them) needed a hard reset, to work for a couple of hours, and then needed a reset again.
I originally thought the outside temperature drop around 1 november was involved, but i see here that the stable version 1.23B which was on all these sensors, has been replaced by some other version over OTA. Maybe the problem affects only the sensors that have been updated, and not those that havn't ?
For the record: my own sensor at home (chip-id = 11730516) was installed in june 2018, and it worked flawlessly with a WiFi signal of -90 dBm up until end october, begin november 2019 (see graphics).
My job as a volunteer is assembling - installing - repairing the 2000+ sensors that our organisation "Luchtpijp.be - Beweging.net - CM" distributes in a wide area around Brussels/Belgium & in Flanders. We even supply other groups with preassembled sensors, so maybe Tavas666 uses our sensors ?

[RikDrabs]

Maybe the solution for now is just reflash with version 1.23B (the stable version of december 2018) and then just turn auto-update off. Or can the 1.23B version be put in OTA, for "downgrading" the firmware, to prevent a manual intervention on 2000+ sensors here in Belgium ?

[RikDrabs]

Maybe the problem is coupled to the WiFi signal value: previously a value of -90 dBm was fine, and the connection worked flawlessly. In the past i even saw sensors working fine at -95, -96 dBm.
Now apparently a value of at least -80 dBm is necessary (according to what you say)
So every sensor which had a signal below -80 dBm in the past, and worked perfectly with that signal, is now in trouble with the new update.

[dirkmueller]

@RikDrabs the reason both problems appeared on oktober 31st is because that's when the first stable OTA rolled out in 2019.

So you're saying thetr is a problem in that build. I hear you. Let's find out what the issue is rather than flashing an old version and turning update off.

Can you pick one of the sensor that exhibits the problem and turn 'use beta version' in config on? Does that improve the situation? Beta is about 100 commits ahead of stable s already with a large upgrade of the base os firmware.

Also can you please share which sensors are chosen here and all the settings other than wifi that you changed from the default.

[Tavas666]

@RikDrabs the sensors I was talking about our not from the 'Luchtpijp' batch (but we do have a lot of colleagues running them but I haven't heard from any problems with them).

Some of the sensors that are causing most of the problems are these:

https://maps.luftdaten.info/grafana/d-solo/000000004/single-sensor-view?orgId=1&panelId=2&var-node=22587&from=1573407851684&to=1573721020012

https://www.madavi.de/sensor/signal.php?sensor=esp8266-5940021

https://maps.luftdaten.info/grafana/d-solo/000000004/single-sensor-view?orgId=1&panelId=2&var-node=22591&from=1573407851684&to=1573721020012

https://www.madavi.de/sensor/signal.php?sensor=esp8266-993863

https://maps.luftdaten.info/grafana/d-solo/000000004/single-sensor-view?orgId=1&panelId=2&var-node=22593&from=1573407851684&to=1573721020012

https://www.madavi.de/sensor/signal.php?sensor=esp8266-5940551

As I wrote all of these need frequent hard resets to start resending data. They all use a local MIFI network and I suspect the signal strength can sometimes drop but in the past this never stopped sensors from operating.

The one I have here at home now is the first one above. I have turned the beta-firmware option on now (>>> this info is corrected, I first changed the option on another sensor at home. The problematic sensor had the auto update box checked).

As for the others, they are all co-located with our reference stations so spread out over the country. I have no way to access them in short notice.

[dirkmueller]

Great, as long as we have one to reproduce that's enough. I can look for a set of patches to revert to understand where the issue comes from.

[Tavas666]

Just to clarify, I did have 2 sensors mixed up...the one causing problems did have the auto update box checked. It now has the beta firmware also checked and after a hard reset it's working again...let's see how it goes >>
https://www.madavi.de/sensor/graph.php?sensor=esp8266-5940021-sds011

[dirkmueller]

Looks like platformio/platform-espressif8266#180 (comment) might be giving a clue..

[Tavas666]

Small update, the sensor with the new beta is still up and running. The others have stopped at least once in the last 24h.

[Tavas666]

Two days ago I fysically went to all the sensors that were causing trouble. I ticked the beta firmware boxes and had no problems since them...so that appears to have solved the problem.

[dirkmueller]

Thanks for confirming that the issue is already fixed in beta tree. That's great Happy Friday news!

[RikDrabs]

@dirkmueller: I was temporarily unavailable (for medical reasons), but i did what you asked previously: i've put the beta flag on in my sensor 11730516: it started working normally again. It's an extra confirmation that the problem is the same, and that it is solved in the beta branch. Thanks !
When will this solution be propagated to the other sensors in trouble? Or is this not possible, because of the bad WiFi connection caused by the problem? My question: do we have to visit each and every sensor in trouble and repair them manually, or will this problem be solved automatically ?
And if automatic repair is possible, how much time before the next firmware release ? (to tell our users)

[dirkmueller]

I hope we can push out a new stable release before December 6th. Currently in beta testing, when there are no new issues reported, we'll release that as stable.